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Executive  Summary 


Background 

Leaders  in  the  DoD  analysis  community  are 
calling  for  an  examination  of  how  we  perform 
analyses.  They  are  asking  us  whether  our 
analysis  practices  warrant  examination  with  a 
view  towards  improving  their  effectiveness. 

The  management  of  knowledge,  how  to  prepare 
to  perform  the  next  analysis,  how  to  perform  the 
next  analysis,  how  to  gather  information  needed 
to  support  analysis,  and  how  to  find  and  apply 
analytic  insights  from  past  studies,  are  becoming 
principal  factors  in  being  able  to  meet  decision 
makers’  expectations  in  regard  to  both  the 
quality  of  an  analysis  and  the  speed  with  which 
an  analysis  is  accomplished. 

"Making  Analysis  Relevant"  was  the  topic  of  a 
PHALANX  article  by  Vince  Roske  in  early 
1998.1  Mr  Roske  reiterated  a  recipe  provided  by 
John  D.  (Dave)  Robinson,  MG  USA,  Ret,  when 
he  was  the  Director  of  J-8.  The  recipe  is 
provided  below: 

•  What ’s  the  question  ? 

•  What's  the  “real”  question? 

•  What  do  the  final  slides  look  like? 

•  What  do  I  already  know  ? 

•  How  do  I  get  the  remaining  information  that 
I  need? 

Dave  Robinson’s  recipe  is  reminiscent  of  the 
practical  battle  drills  used  by  the  military  to  train 
soldiers  to  do  basic  tasks.  It  is  simple,  it  is 
memorable,  and  it  will  get  the  job  done.  Mr 
Roske  goes  on  to  look  deeper  at  each  of  the 
recipe  steps  and  discusses  the  issues  associated 
with  performing  analysis  today.  He  explains 
that  we  must  focus  on  the  purpose  of  the 
analysis  and  not  become  wed  to  models  that 
don’t  fit  the  analysis.  Furthermore,  he  explains 
what  the  analyst  must  do  to  keep  the  analysis 
relevant  to  the  decision  maker. 


Discussion 

The  keys  (and  related  impediments)  to  effective 
analysis  in  DoD  are: 

The  analysts, 

•  Turnover  and  reduction  in  numbers  of 
analyst  positions  in  DoD  organizations  is 
creating  a  critical  situation. 

The  knowledge  that  analysts  access, 

•  DoD  analysts  are  spread  around  the  world 
and  find  it  very  difficult  to  collaborate. 

The  process  by  which  analysts  perform  their 
work, 

•  DoD  does  not  have  a  best  practices  guide  to 
aid  analysts  in  their  work. 

The  tools  that  analysts  employ  to  perform  their 
work, 

•  The  analytical  methods  and  tools  employed 
by  DoD  analysts  are  increasingly  complex 
as  are  the  problems  to  which  the  methods 
and  tools  are  being  applied. 

The  effectiveness  with  which  analysts 
communicate  analytical  results  to  decision 
makers, 

•  The  quantity,  variety  and  complexity  of 
military  operations  is  increasing  —  the  time 
to  perform  analysis  and  communicate  results 
to  decision  makers  is  decreasing. 

The  proper  management  of  knowledge  can 
improve  analytical  efficiency  by  providing  a 
framework,  tools  and  techniques  to  reuse 
intellectual  assets.  By  marshalling  resources  to 
respond  to  opportunities  and  threats,  our 
responsiveness  could  be  vastly  improved.  And 
by  bringing  people  together  across  time  and 
geography  to  share  ideas  and  insight,  intellectual 
innovation  can  flourish  and  bear  richer  insight 
for  decision  makers. 


1  Mr  Vincent  P.  Roske,  Jr.,  “Making  Analysis  Relevant,’ 
PH/4L4/VX  Volume  31,  Number  1,  March  1998 
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This  special  meeting,  comprising  analysts  and 
subject  matter  experts  in  approaches  to  analysis, 
concentrated  on  the  improvement  of  the  analysis 
practice  throughout  DoD.  To  achieve  this  end, 
participants  enumerated  potential  issues  related 
to  a  broad  move  towards  more  collaborative 
analysis  and  enterprise  knowledge  management 
and  define  the  appropriate  policies, 
organizational  structure,  processes,  technologies 
and  training  needed  to  institutionalize  an 
improved  practice  of  analysis  in  DoD.  These 
issues  were  addressed  by  Working  Groups 
(WGs),  addressing  the  spectrum  of  dimensions 
of  organizational  change,  as  well  as  supporting 
infrastructures  and  supporting  tools  and 
techniques. 

Goals  and  Objectives 

The  body  of  analytical  methods,  techniques  and 
tools  and  how  we  apply  them  was  the  focus  of 
this  workshop.  We  examined  how  we  carry  out 
our  professional  work  and  the  need  to  evolve. 
Questions  addressed  included: 

•  What  should  be  the  customary  manner  for 
performing  defense-related  analyses? 

•  How  do  we  manage  the  knowledge  we 
derive  from  our  analyses? 

•  How  do  we  support  both  fast  turn-around 
and  more  deliberate  analyses? 

•  What  problems  are  not  well  served  by  the 
current  state  of  the  practice? 

•  What  innovative  developments  in  analysis 
tools  and  methods,  including  Knowledge 
Management,  hold  potential  value  for  DoD 
analysis? 

Goals 

This  workshop  afforded  the  military  operations 
research  community  an  opportunity  to  achieve 
the  following  goals: 


•  A  vision  for  improving  analytic  practice  and 
the  management  of  supporting  analytic 
knowledge  in  DoD. 

•  A  plan  to  realize  the  vision  to  improve 
analytic  practice  and  the  management  of 
supporting  analytic  knowledge  in  DoD. 

•  A  definition  of  how  Military  Operations 
Research  Society  (MORS)  may  work  with 
the  Sponsors  over  time  to  implement 
improvements  to  analytic  practice  and  to  the 
management  of  analytic  knowledge  in  DoD. 

Objectives 

The  objectives  were  to: 

•  Define  a  desired  analytic  practice  approach 
in  DoD  in  terms  of  policy,  organization, 
process,  technology  and  training. 

•  Define  current  analytic  practice  in  DoD  in 
terms  of  policy,  organization,  process, 
technology  and  training. 

•  Identify  weaknesses  in  the  current  practice 
relative  to  current  and  future  requirements. 

•  Produce  an  actionable  plan  to  move  DoD  to 
the  desired  state. 

•  Define  mechanisms  to  be  established  within 
MORS  to  work  with  the  DoD  Sponsors  to 
execute  the  plan. 

Approach 

In  order  to  achieve  these  goals  and  objectives, 
the  subject  of  analytic  practice  was  examined  in 
a  number  of  different  contexts.  Working  groups 
examined  analytic  practice  within  each 
contextual  framework.  WGs  were  asked  to 
address  some  baseline  issues  (as  a  starting  point) 
and  provide  specific  results.  The  workshop 
sessions  are  listed  in  Table  E-l. 


•  A  shared  understanding  of  the  need  to 
evolve  the  practice  of  military  operations 
analysis  in  DoD. 
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Sessions 

Topics 

Leadership 

Keynote 

Session 

Dr  Ruth  David: 

Knowledge 

Management 

Jim  Johnson: 
Perspectives  on 
Requirements 
Analysis 

Bruce  Powers: 
Perspectives  on 
Capabilities 
Analysis 

Workshop  Lead 
Proponent: 

Vince  Roske 

Workshop  Chair: 
Denis  Clements 

Workshop  Co- 
Chair:  Sue 

Iwanski 

Plenary 

Sessions 

Mike  Yoemans: 
Enterprise 
Knowledge 
Management 

Dr  Tom  Allen: 

Joint  Analysis 

QDR  2001  and 
Beyond 

Working 

Groups 

Management  of 
Analytic 

Knowledge 

WG  Chair:  Mike 
Yoemans 

WG  Co-Chair: 
Gary  Coe 

Enabling 

Collaboration 

WG  Chair:  COL 
Bob  Clemence 

WG  Co-Chair: 

Lt  Col  Kirk  Yost 

New  Analysis 
Challenges 

WG  Chair:  Dr 
Steve  Pilnick 

WG  Co-Chair: 

Lt  Col  Suzanne 
Beers 

New  Analysis 

Tools  and 

Methods 

WG  Chair:  LTC 
Dan  Maxwell 

WG  Co-Chairs: 
John  Furman  and 
Dr  A1  Brandstein 

Synthesis 

Group 

Consolidating 
Recommendations 
and  Developing 
an  Action  Plan 

Group  Chair:  Dr 
Stuart  Starr,  FS 
Group  Co- 
Chairs:  Dr  Jerry 
Kotchka,  Dr 

Tom  Allen 

Table  E-l.  Workshop  Organization 


Working  Group  Sessions 

The  workshop  attendees  were  organized  into 
four  subject  area  WGs  of  about  20  to  30 
members  each  plus  a  Synthesis  Group  to  address 
the  overall  goals  and  objectives. 


WG  1,  Management  of  Analytic  Knowledge, 
examined  the  manner  in  which  analytic 
knowledge  is  discovered,  expanded,  organized 
and  shared. 

WG  2,  Enabling  Collaboration,  examined  the 
process,  tools  and  techniques  that  support 
distributed  analytic  teams  in  performing  analytic 
tasks. 

WG  3,  New  Analysis  Challenges,  examined 
military  operations  analysis  questions  of  interest 
for  which  the  established  military  operations 
analysis  community  apparently  does  not  have 
answers. 

WG  4,  New  Analysis  Tools  and  Methods, 
examined  emerging  developments  in  analytical 
approaches  to  complex  systems  to  assess 
potential  applications  to  challenging  military 
operations  analysis  problems. 

The  Synthesis  Group  sought  to  discover  the 
common  themes  that  tie  together  the  efforts  of 
the  four  subject  area  WGs  and  provide  feedback 
to  the  groups  on  a  continuing  basis. 

Workshop  Synopsis 

Figure  E-l  shows  an  illustration  of  an  ideal  end- 
state  for  the  current  efforts  to  improve  analytical 
practices.  There  are  three  families  of  models  and 
simulations:  M&S  embedded  in  real-time 
command  and  control  systems;  training  and 
exercise  support  simulations;  and  models  and 
simulations  used  to  support  analysis  (to  include 
Test  and  Evaluation  (T&E)  analysis  and  analysis 
of  non-warfighting  issues  such  as  infrastructure 
and  business  process  improvement).  Within  each 
domain  are  a  set  of  models  and  simulations 
tailored  to  the  purpose  at  hand.  No  single  model 
will  apply  universally  across  all  domains,  but  in 
this  ideal  end  state  we  have  data,  software 
objects  and  process  representations  shared 
across  the  domains  where  it  is  logically 
consistent  with  the  model  and  purpose. 
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Figure  E-l.  Potential  Future  “Framework” 

Infrastructure  programs  currently  under 
development  are  assumed  to  be  in  place  in  this 
end  state,  to  include  the  High-Level  Architecture 
(HLA)  for  simulations  and  the  DII  COE  linking 
all  areas  together.  HLA  federations,  which  may 
include  simulations  widely,  distributed  over 
space,  will  exist  where  appropriate,  but  these 
federations  should  be  standardized  and  validated 
for  consistency  across  federates  before  use.  The 
creation  of  “on  the  fly”  federations  that  grab 
objects  and  federates  scattered  across  the  DII  to 
create  unique  environments  is  probably  still  not 
possible  in  this  end  state  because  there  is  no 
guarantee  that  the  representations  will  be 
internally  consistent  (consistency  is  not  assured 
even  if  data  is  passed  through  HLA),  and  it  is 
unlikely  that  a  single  analyst  or  team  located  at  a 
single  site  will  be  familiar  with  all  of  the  objects 
or  federates  potentially  available.  Instead,  these 
simulations  may  be  combined  through  the  use  of 
collaborative  analysis  rather  than  software 
linkages. 

All  of  the  tools  are  presumed  to  be  in  a  bed  of 
warm  data.  That  is,  there  is  a  data  infrastructure 
that  makes  data  readily  available  (with 
exhaustive  metadata  explaining  the  derivation 
and  accredited  use  of  the  data)  for  all  of  these 
systems.  For  example,  an  analysis  of  real-world 
capability  should  be  able  to  draw  on  the  same 
data  set  showing  current  status  and  deployments 
of  units  that  a  real-time  command  and  control 
system  might  use.  Similarly,  an  analysis  of  a 
future  capability  (for  example,  a  2010  force  in  a 


particular  scenario)  should  be  able  to  draw  on  a 
validated  and  commonly  used  set  of  data  to 
represent  the  scenario,  system  performance, 
OPFOR  performance,  etc.  Generation  and 
storage  of  this  data  requires  a  significant 
resource  commitment  within  DoD,  but  it  is 
necessary  to  achieve  the  desired  end  state. 

A  key  feature  of  this  environment  is  that  the 
output  of  the  models,  as  well  as  the  results  of 
specific  analyses  and  the  lessons  learned  from 
training  and  exercises  are  also  added  to  the 
common  database.  This  provides  a  useful 
source  of  information  that  can  be  of  particular 
help  to  quick  turn-around  analyses,  which  may 
synthesize  the  outputs  of  previous  models, 
exercises  and  analyses. 

The  Action  Plan 

The  Synthesis  Group  identified  a  number  of 
actions  that  should  become  part  of  a  long-range 
action  plan  that  the  community  should  consider 
and  flesh  out  over  the  next  few  years  as  it 
evolves  the  practice  of  military  analysis  in  DoD. 

There  are  several  steps  that  MORS  can 
implement  immediately  that  have  little  work 
impact  but  enormous  potential  to  improve  the 
ability  of  this  community  to  connect  with  each 
other  and  better  meet  the  over  all  needs  of  its 
membership.  In  addition  to  the  items  listed 
below,  the  working  groups  provided  a  number  of 
general  and  specific  steps  that  will  help  the 
community  learn  to  harness  knowledge 
management  and  collaboration,  as  well  as  to 
address  emerging  issues  in  the  changing  security 
environment.  Generalized  recommendations 
endorsed  by  the  workshop  include  working  the 
people,  process  and  technology  problems  in 
order  to  ensure  a  better  final  product. 

Actions  for  MORS  include: 

•  Near  Term: 

Update  the  MORS  membership  directory 
and  database  with  more  information  on  each 
member. 

Evolve  the  role  of  the  Electronic  Media 
Committee  to  address  enterprise  knowledge 
management. 
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Contribute  to  a  Code  of  Best  Practices  for 
Analysis  and  Joint  Experimentation. 

•  Mid  Term: 

Undertake  focused  workshop  activity  on  this 
topic. 

•  Far  Term: 

Use  the  Education  Colloquia  to  address 
ideas  that  Knowledge  Management  is  an 
important  element  of  professional 
development. 

Actions  for  DoD  include: 

The  working  group  recommendations 
provide  a  phased  approach  to  knowledge 
management  and  collaboration 
implementation  by  focusing  on  people, 
process,  and  technology  and  product,  as  well 
as  list  challenges  and  promising  tools  and 
methodologies  to  meet  those  challenges. 

Table  E-2  summarizes  the  focus  and  potential 
impact  of  the  specific  actions  recommended. 
First,  cultural  changes  are  a  necessary 
prerequisite  to  effective  transformation  of  the 
analysis  community  (e.g.,  the  need  to  dispel  the 
myth  that  “knowledge  is  power”).  Second  are 
several  organizational  issues  that  must  be 
addressed.  One  of  these  is  to  ensure  top 
management  buys  in  to  this  transformation. 
Another  includes  reassessing  how  security  issues 
can  be  addressed  in  an  information-centric,  open 
environment.  In  the  third  area,  decision-makers 
must  recognize  that  the  analysts  involved  in  all 
phases  of  the  assessment  process  constitute  a 
critical  intellectual  resource. 


_ Summary  Recommendations _ 

Vision/Policy:  Develop  and  promulgate  an  Enterprise 

Vision  and  strategic  directions. _ 

Culture:  Take  action  to  remove  deeply  rooted 
cultural  barriers  (e.g.,  remove  the  incentives  to  hoard 

knowledge) _ 

Organization:  Ensure  that  there  is  buy-in  by  Top 
Management.  Investigate  organizational  constructs 
including  business  rules  that  best  enable  global 

collaboration  of  military  analysts. _ 

People:  Provide  incentives  to  analysts  to  collaborate 

and  share  knowledge. _ 

Process:  Institutionalize  processes  to  enhance 
collaboration  and  the  sharing  of  tacit  and  explicit 

knowledge  among  the  analytical  community. _ 

Analytic  Tools: 

•  Develop  and  emphasize  joint  standards  in 
algorithm  and  data  development. 

•  Expand  the  development  and  maintenance  of 
data  warehouses. 

•  Focus  basic  research  and  tool  development  to 

_ meet  emerging  challenges _ 

Table  E-2.  Summary  Recommendations 

They  must  be  given  incentives  to  stimulate  them 
to  contribute  and  draw  from  future  knowledge 
warehouses.  Tying  all  this  together  are  the  links 
between  the  vision  and  the  high  level  guidance 
to  the  analysis  process  and  the  primary  products 
of  that  process  (i.e.,  effective,  timely,  rigorous 
analyses).  To  implement  this  linkage,  a  set  of 
enhanced  processes  is  needed  (e.g., 
institutionalized  processes  to  diffuse  tacit  and 
explicit  knowledge)  supported  by  a  broad  set  of 
tools  (e.g.,  knowledge  warehouses,  collaboration 
tools).  These  processes  and  tools  are  constrained 
by  available  resources  (e.g.,  funds,  manpower) 
such  that  a  special  funding  mechanism  may  be 
required  to  satisfy  time-urgent  joint  needs.  In 
particular,  basic  research  must  be  continued  in 
those  areas  where  understanding  of 
phenomenology  is  not  available  (human  decision 
processes,  Information  Operations/Information 
Warfare  (IO/IW),  etc)  as  well  as  the  continuing 
development  of  analysis  tools  to  address  the 
emerging  challenges. 
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The  Synthesis  Group  feels  that  by  addressing  the 
needs  of  the  community  to  implement  our 
vision,  the  practice  of  analysis  can  evolve  from 
its  current  position  of  a  limited  toolkit  and 
methodologies  to  address  a  limited  number  of 
issues  to  a  complete  tool  set  connecting  analysts 
and  decision  makers  to  address  the  full  array  of 
national  security  challenges. 

Summary 

This  workshop  has  demonstrated  that  there  is 
great  potential  to  be  realized  from  the  harnessing 
of  the  knowledge  in  our  heads  and  in  the 
systems  that  support  our  analytical  efforts.  In 
addition,  there  is  great  power  in  undertaking  a 
more  collaborative  approach  to  the  analyses  that 
we  perform.  The  combination  of  enterprise 
knowledge  management  and  collaboration  can 
deliver  better  analysis  products  faster. 


The  way  ahead  to  realize  this  vision  will  be 
difficult.  Nothing  less  than  a  cultural  change  is 
required.  The  demands  of  the  national  security 
environment  require  that  we  find  ways  to  alter 
the  information  security  environment  to  permit 
knowledge  sharing  and  collaboration  on  an 
enterprise  scale.  Getting  analysts  to  collaborate 
and  breaking  down  the  barriers  to  that 
collaboration  and  knowledge  sharing  are 
fundamental  changes  that  require  bottom  up  as 
well  as  top  down  change. 

The  members  of  this  workshop  acknowledge  the 
potential  value  of  such  change.  It  is  now  up  to 
us  to  obtain  the  buy-in  of  the  DoD  leadership  by 
convincing  them  that  the  benefits  are  worth  the 
investment  of  time,  money  and  energy  to  realize 
the  vision  articulated  by  this  workshop. 
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Workshop  Report 

Evolving  the  Practice  of  Military  Operations  Analysis  in  DoD 
Applying  Knowledge  Management 


Background 

Leaders  in  the  DoD  analysis  community  are 
calling  for  an  examination  of  how  we  perform 
analyses.  They  are  asking  us  whether  our 
analysis  practices  warrant  examination  with  a 
view  towards  improving  their  effectiveness. 

The  management  of  knowledge,  how  to  prepare 
to  perform  the  next  analysis,  how  to  perform  the 
next  analysis,  how  to  gather  information  needed 
to  support  analysis,  and  how  to  find  and  apply 
analytic  insights  from  past  studies,  are  becoming 
principal  factors  in  being  able  to  meet  decision 
makers’  expectations  in  regard  to  both  the 
quality  of  an  analysis  and  the  speed  with  which 
an  analysis  is  accomplished. 

"Making  Analysis  Relevant"  was  the  topic  of  a 
PHALANX  article  by  Mr  Vince  Roske  in  early 
1998.2  Mr  Roske  reiterated  a  recipe  provided  by 
John  D.  (Dave)  Robinson,  MG  USA,  Ret,  when 
he  was  the  Director  of  J-8.  The  recipe  is 
provided  below: 

•  What's  the  question  ? 

•  What's  the  “real”  question? 

•  What  do  the  final  slides  look  like? 

•  What  do  I  already  know  ? 

•  How  do  I  get  the  remaining  information  that 
I  need? 

Dave  Robinson’s  recipe  is  reminiscent  of  the 
practical  battle  drills  used  by  the  military  to  train 
soldiers  to  do  basic  tasks.  It  is  simple,  it  is 
memorable  and  it  will  get  the  job  done.  Mr 
Roske  goes  on  to  look  deeper  at  each  of  the 
recipe  steps  and  discusses  the  issues  associated 
with  performing  analysis  today.  He  explains 
that  we  must  focus  on  the  purpose  of  the 
analysis  and  not  become  wed  to  models  that 


don’t  fit  the  analysis.  Furthermore,  he  explains 
what  the  analyst  must  do  to  keep  the  analysis 
relevant  to  the  decision  maker. 

Discussion 

The  first  three  steps  in  General  Robinson’s 
recipe  address  critical  steps  in  organizing  and 
focusing  an  analysis  or  study.  Answering 
“ What's  the  question ?”  and  “ What's  the  “real” 
question?”  enables  the  analyst  to  frame  a  study 
so  that  there  is  a  chance  that  it  may  ultimately  be 
relevant  to  the  needs  of  the  decision  maker. 
Answering  “ What  do  the  final  slides  look  like?” 
up  front  serves  the  dual  purpose  of  stressing  the 
importance  of  communicating  results  to  the 
decision  maker  and  focusing  the  analysis  efforts 
from  the  beginning. 

But  the  substantive  content  of  the  analysis  is 
contained  in  the  last  two  recipe  steps:  “What  do  I 
already  know?”  and  “How  do  I  get  the 
remaining  information  I  need?”  Evolving  the 
practice  of  military  operations  analysis  in  DoD 
must  therefore  substantively  address  these  last 
two  questions. 

One  of  the  keys  to  successfully  evolving  the 
practice  of  analysis  is  the  innovative 
management  and  application  of  knowledge,  both 
tacit  (the  knowledge  in  the  minds  of  the  experts) 
and  explicit  (the  knowledge  that  is  recorded  in 
various  ways).  There  is  an  explosion  of  new 
thinking  and  new  supporting  technology  to 
capture  and  apply  knowledge  more  effectively. 
This  area  includes  innovation  in  the  discovery, 
expansion,  capture,  organization  and  sharing  of 
knowledge.  Innovations  include:  correlation 
landscapes,  data  mining  tools,  virtual 
information  centers  and  influence  diagrams. 


2  Mr  Vincent  P.  Roske,  Jr.,  “Making  Analysis  Relevant,” 
PHALANX  Volume  31,  Number  1,  March  1998 


1 


The  keys  (and  related  impediments)  to  effective 
analysis  in  DoD  are: 

The  analysts, 

•  Turnover  and  reduction  in  numbers  of 
analyst  positions  in  DoD  organizations  is 
creating  a  critical  situation. 

The  knowledge  that  analysts  access, 

•  DoD  analysts  are  spread  around  the  world 
and  find  it  very  difficult  to  collaborate. 

The  process  by  which  analysts  perform  their 
work, 

•  DoD  does  not  have  a  best  practices  guide  to 
aid  analysts  in  their  work. 

The  tools  that  analysts  employ  to  perform  their 
work, 

•  The  analytical  methods  and  tools  employed 
by  DoD  analysts  are  increasingly  complex 
as  are  the  problems  to  which  the  methods 
and  tools  are  being  applied. 

The  effectiveness  with  which  analysts 
communicate  analytical  results  to  decision 
makers, 

•  The  quantity,  variety  and  complexity  of 
military  operations  is  increasing  —  the  time 
to  perform  analysis  and  communicate  results 
to  decision  makers  is  decreasing. 

Information  is  a  source  for  good  decision 
making,  however,  a  piece  of  information  is  more 
valuable  when  presented  in  context,  that  is,  with 
other  pieces  of  related  information  that  makes 
the  original  information  more  meaningful  and 
actionable.  For  instance,  a  study  of  the 
effectiveness  of  a  proposed  Intelligence, 
Surveillance  and  Reconnaissance  (ISR)  system 
may  provide  important  information,  but  it  is 
much  more  meaningful  and  actionable  when 
analyzed  in  the  context  of  national  security 
policy  and  DoD  objectives  and  when  related  to 
operational  outcomes. 

In  DoD,  we  analysts  don’t  know  what  we  know, 
and  consequently,  we  are  bound  to  reinvent  the 
wheel  many  times.  We  do  not  have  automated 
enterprise  repositories  which  hold  accessible, 


important,  explicit  knowledge  nor  do  we  have 
the  means  to  locate  the  right  subject  matter 
experts  quickly  and  easily  to  access  their  tacit 
knowledge. 

The  proper  management  of  knowledge  can 
improve  analytical  efficiency  by  providing  a 
framework,  tools  and  techniques  to  reuse 
intellectual  assets.  By  marshalling  resources  to 
respond  to  opportunities  and  threats,  our 
responsiveness  could  be  vastly  improved.  And 
by  bringing  people  together  across  time  and 
geography  to  share  ideas  and  insight,  intellectual 
innovation  can  flourish  and  bear  richer  insight 
for  decision  makers. 

This  special  meeting,  comprising  analysts  and 
Subject  Matter  Experts  (SMEs)  in  approaches  to 
analysis,  concentrated  on  the  improvement  of 
the  analysis  practice  throughout  DoD.  To 
achieve  this  end,  participants  enumerated 
potential  issues  related  to  a  broad  move  towards 
more  collaborative  analysis  and  enterprise 
knowledge  management  and  define  the 
appropriate  policies,  organizational  structure, 
processes,  technologies  and  training  needed  to 
institutionalize  an  improved  practice  of  analysis 
in  DoD.  These  issues  were  addressed  by 
working  groups,  addressing  the  spectrum  of 
dimensions  of  organizational  change,  as  well  as 
supporting  infrastructures  and  supporting  tools 
and  techniques. 

Goals  and  Objectives 

The  body  of  analytical  methods,  techniques  and 
tools  and  how  we  apply  them  was  the  focus  of 
this  workshop.  We  examined  how  we  carry  out 
our  professional  work  and  the  need  to  evolve. 

Questions  addressed  included: 

•  What  should  be  the  customary  manner  for 
performing  defense-related  analyses? 

•  How  do  we  manage  the  knowledge  we 
derive  from  our  analyses? 

•  How  do  we  support  both  fast  tum-around 
and  more  deliberate  analyses? 
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•  What  problems  are  not  well  served  by  the 
current  state  of  the  practice? 

•  What  innovative  developments  in  analysis 
tools  and  methods,  including  knowledge 
management,  hold  potential  value  for  DoD 
analysis? 

Goals 

This  workshop  afforded  the  military  operations 

research  community  an  opportunity  to  achieve 

the  following  goals: 

•  A  shared  understanding  of  the  need  to 
evolve  the  practice  of  military  operations 
analysis  in  DoD. 

•  A  vision  for  improving  analytic  practice  and 
the  management  of  supporting  analytic 
knowledge  in  DoD. 

•  A  plan  to  realize  the  vision  to  improve 
analytic  practice  and  the  management  of 
supporting  analytic  knowledge  in  DoD. 

•  A  definition  of  how  Military  Operations 
Research  Society  (MORS)  may  work  with 
the  Sponsors  over  time  to  implement 
improvements  to  analytic  practice  and  to  the 
management  of  analytic  knowledge  in  DoD. 

Objectives 

The  objectives  were  to: 

•  Define  a  desired  analytic  practice  approach 
in  DoD  in  terms  of  policy,  organization, 
process,  technology  and  training. 

•  Define  current  analytic  practice  in  DoD  in 
terms  of  policy,  organization,  process, 
technology  and  training. 

•  Identify  weaknesses  in  the  current  practice 
relative  to  current  and  future  requirements. 

•  Produce  an  actionable  plan  to  move  DoD  to 
the  desired  state. 

•  Define  mechanisms  to  be  established  within 
MORS  to  work  with  the  DoD  Sponsors  to 
execute  the  plan. 


Approach 

In  order  to  achieve  these  goals  and  objectives, 
the  subject  of  analytic  practice  was  examined  in 
a  number  of  different  contexts.  Working  groups 
examined  analytic  practice  within  each 
contextual  framework.  WGs  were  asked  to 
address  some  baseline  issues  (as  a  starting  point) 
and  provide  specific  results.  The  workshop 
sessions  are  listed  in  Table  1. 


Sessions 

Topics 

Leadership 

Keynote 

Session 

Dr  Ruth  David: 

Knowledge 

Management 

Jim  Johnson: 
Perspectives  on 
Requirements 
Analysis 

Bruce  Powers: 
Perspectives  on 
Capabilities 

Analysis 

Workshop  Lead 
Proponent:  Vince 
Roske 

Workshop  Chair: 
Denis  Clements 

Workshop  Co- 
Chair:  Sue 

Iwanski 

Plenary 

Sessions 

Mike  Yoemans: 
Enterprise 
Knowledge 
Management 

Dr  Tom  Allen: 

Joint  Analysis 

QDR  2001  and 
Beyond 

Working 

Groups 

Management  of 
Analytic 

Knowledge 

WG  Chair:  Mike 
Yoemans 

WG  Co-Chair: 

Gary  Coe 

Enabling 

Collaboration 

WG  Chair:  COL 
Bob  Clemence 

WG  Co-Chair:  Lt 
Col  Kirk  Yost 

New  Analysis 
Challenges 

WG  Chair:  Dr 

Steve  Pilnick 

WG  Co-Chair:  Lt 
Col  Suzanne 

Beers 

New  Analysis 

Tools  and  Methods 

WG  Chair:  LTC 
Dan  Maxwell 

WG  Co-Chairs: 
John  Furman  and 

Dr  AlBrandstein 

Synthesis 

Group 

Consolidating 
Recommendations 
and  Developing  an 
Action  Plan 

Group  Chair:  Dr 
Stuart  Starr,  FS 
Group  Co-Chairs: 
Dr  Jerry  Kotchka, 
Dr  Tom  Allen 

Table  1.  Workshop  Organization 
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Plenary  Session  Presentations 

The  Keynote  speaker  was  Dr  Ruth  David,  CEO 
of  ANSER  a  Federally  Funded  Research  and 
Development  Center  (FFRDC),  who  discussed 
“Enterprise  Knowledge  Management.”  In  the 
context  of  a  hypothetical  story  with  the  entire 
national  security  apparatus  as  the  enterprise  and 
Homeland  Defense  as  the  strategic  objective,  she 
explained  that  in  the  21st  century  our  nation’s 
security  would  depend  on  effective  knowledge 
management.  Knowledge  management  was 
defined  based  on  the  article  “On  Knowledge 
Management”  by  Ramon  Barquin  in  the 
November/December  1999  issue  of  the  E-Gov 
Journal :  “Knowledge  Management  is  the 
process  through  which  an  enterprise  uses  its 
collective  intelligence  to  accomplish  its  strategic 
objectives.” 

Dr  David  quoted  Bran  Ferren,  President  of  Walt 
Disney  Imagineering  who  stated  that  no  “big 
idea”  would  survive  a  requirements  process. 

She  asserted  that  Enterprise  Knowledge 
Management  is  definitely  a  big  idea,  which 
requires  the  reinvention  of  the  way  we  operate 
and  that  our  requirements’  processes  and 
Program  Objective  Memorandum  (POM)  build 
cycles  will  not  effectively  support  this.  To 
successfully  achieve  this  we  need  to  focus  on  the 
four  key  elements  critical  to  the  success  of 
Knowledge  Management:  Strategy,  People, 
Process  and  Technology. 

The  strategy  provides  the  motivation,  or  sense  of 
purpose  for  the  activity.  Effective  knowledge 
management  requires  a  shared  goal  or  objective 
—  and  the  strategy  describes  the  way  that  goal 
will  be  achieved.  People  are  the  next  critical 
element.  Without  people  who  create  and  use  the 
knowledge,  the  whole  issue  is  moot.  People,  or 
participants  in  the  enterprise,  must  be  motivated 
to  share  not  only  information,  but  knowledge  as 
well.  That  is,  they  must  do  more  than  simply 
populate  databases  and  other  static  repositories. 
They  must  actively  and  personally  engage  in 
building  and  using  the  collective  intelligence  of 
the  enterprise.  And,  these  participants  must  span 
the  spectrum  from  those  who  collect  and  analyze 
information  to  those  who  take  action  based  on 
the  results.  That  is,  the  traditional  “customers” 


of  the  analytic  community  —  whether  they  are 
warfighters,  policy  makers,  law  enforcement 
officials  or  local  first  responders  —  must  be 
active  participants,  expanding  the  collective 
intelligence  of  the  enterprise. 

In  terms  of  process,  if  Knowledge  Management 
is  defined  as  the  overall  process,  then  critical 
sub-processes  include  cooperation, 
communication  and  collaboration.  Michael 
Schrage  in  Shared  Minds:  The  New 
Technologies  of  Collaboration  stated  that  the 
core  processes  must  include  robust  cooperation, 
communication  and  collaboration.  In  his  book, 
Mr  Schrage  defines  cooperation  as  “people 
working  together  for  a  common  purpose  without 
necessarily  having  defined  the  purpose,” 
communication  as  “the  exchange  of 
information”  and  collaboration  as  “an  act  of 
shared  creation  and/or  shared  discovery.” 

Finally,  we  must  have  the  enabling  technologies 
to  make  connections,  from  people  to  machines, 
between  machines  and  among  people. 
Computation  can  be  quite  powerful,  but  it  is 
fundamentally  stupid!  Emerging  adaptive 
algorithms  hold  the  potential  for  moving  from 
“what  you  ask  is  what  you  get”  to  “what  you 
need  is  what  you  get”  in  searching  for 
information.  This  type  of  smart  search  has  the 
potential  to  save  significant  time  and  effort. 

Mr  Jim  Johnson,  Deputy  Director  for  Theater 
Assessments  and  Planning,  OSD,  ODPA&E, 
discussed  “Perspectives  on  Requirements 
Analysis.”  He  emphasized  the  need  to  do  a 
thorough  job  of  analyzing  the  trade  space  to 
understand  not  only  cost  versus  effectiveness  but 
also  the  relationship  of  cost  and  risk. 

Jim  illustrated  that  the  shape  of  the  cost  versus 
effectiveness  curve  in  systems  effectiveness 
analysis,  when  explored  and  well  understood 
will  reveal  where  this  relationship  may  become 
very  important  to  the  analysis.  For  example,  if 
maximum  effectiveness  as  a  function  of  cost  is 
achieved  very  early  in  the  system  effectiveness 
curve  and  the  application  of  added  dollars  has 
little  effect  in  growing  effectiveness,  one  may 
then  focus  on  that  part  of  the  curve  where  dollars 
buy  significant  value. 
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Also  illustrated  was  the  notion  of  buying  down 
risk.  As  in  the  system  effectiveness  case,  risk 
may  be  related  to  investment.  Investments  in 
force  structure  are  made  in  light  of  anticipated 
threats.  US  force  structure  reduces  the  threat  of 
war  or  diminishes  the  risk  of  unacceptable  losses 
if  military  operations  are  undertaken.  Again, 
one  must  explore  the  analysis  space  to 
understand  the  shape  of  the  curve  or  the  surface. 
This  will  allow  the  study  to  identify  the  region 
where  added  investment  ceases  to  diminish  risk 
in  a  meaningful  way. 


To  make  results  relevant  he  provided  the 
following  recipe  aligned  with  Dave  Robinson’s 
questions: 


The  Question 

The  Answer 

What  is  the  question? 

Keep  it  simple. 

What  is  the  real  question  ? 

Don’t  forget  what  you 
know. 

What  do  the  final  slides 
look  like  ? 

Encapsulate  complex 
representations. 

What  do  I  already  know? 

Know  your  audience. 

How  do  I  get  the 
remaining  information  I 
need? 

Be  relevant. 

Table  2.  Making  Results  Relevant 


•  Keep  at  least  one  path  open  toward  each 
operational  goal. 

A  very  interesting  feature  of  the  IWAR/CPAM 
strategy  is  to  ensure  that  total  lifecycle  cost  for 
systems  and  follow-on  systems  is  accounted  for. 
The  Navy  employs  a  far-reaching  planning 
horizon  to  ensure  that  the  Navy:  “Develops 
alternative  force  stmcture  paths  at  the  platform 
level  for  ships,  submarines  and  aircraft  that  meet 
fiscal  constraints.  Each  path  will  attempt  to 
maximize  warfighting  capabilities  while 
balancing  recapitalization,  modernization  and 
operating  costs.” 

Bruce  summarized  by  stating  that  in  building  the 
future  Navy  there  is  a  need  to  assess  what  the 
Navy  will  be  doing  —  how  to  make  it  the  most 
capable.  He  emphasized  that  this  must  be  done 
within  projected  cost  constraints  (e.g.,  zero 
growth)  and  in  light  of  other  considerations  such 
as  maintaining  program  stability,  a  strong 
industrial  base,  etc. 

Mr  Mike  Yoemans,  Chief  Knowledge  Officer, 
OSD,  OASD  (C3I),  presented  an  overview  of 
knowledge  management  during  lunch  on  the 
first  day  of  the  workshop. 


Mr  Bruce  Powers,  Director  for  Requirements 
Analysis,  CNO  Staff,  N81,  provided 
"Perspectives  on  Capabilities  Analysis."  He 
described  the  IWAR/CPAM  process  that  the 
Navy  uses  to  plan  the  way  ahead  so  that 
decisions  made  today  do  not  foreclose  flexibility 
in  the  future. 

Bmce  identified  the  four  primary  components  of 
the  mission  of  the  Deputy  CNO  for  Resources, 
Warfare  Requirements  and  Assessments: 

•  Analyze  uses  of  the  Navy,  including  its 
goals  in  operations,  providing  insight  to 
CNO,  SECNAV  and  DoD. 

•  Identify  programmatic  paths  to  provide 
required  future  capabilities. 

•  Clarify  consequences  of  each  path,  and 
possible  blockages  (e.g.,  funding  and 
technology). 


Mike  described  a  framework  for  managing 
knowledge.  Figure  1  illustrates  this  framework. 


Figure  1.  Epistemological  Architecture 

He  emphasized  the  need  to  focus  on  how 
knowledge  is  constructed  through  a  bottom-up 
process  that  accumulates  facts  and  meanings  in 
order  to  develop  data.  Knowledge  is  derived  by 
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human  beings  who  combine  data  with  context. 
This  may  be  understood  by  using  the  metaphor 
of  a  message.  The  message  provides  context 
and  the  data  provides  the  combination  of  facts 
and  meaning.  Humans  who  take  advantage  of 
the  knowledge  they  are  able  to  accumulate  have 
wisdom  if  they  apply  this  knowledge  in 
meaningful  ways.  This  is  a  rather  simple 
epistemology,  however,  it  does  provide  a  useful 
set  of  definitions  and  semantic  relationships  for 
discussions  of  Knowledge  Management. 

Mike  went  on  to  emphasize  the  need  for  a 
Global  Information  Grid  (GIG)  that  currently 
does  not  exist  as  a  necessity  for  achieving 
information  superiority.  One  of  the  goals  of 
ASD  (C3I)  is  to  promote  the  development  of 
Knowledge  Management  and  a  skill-based 
workforce  throughout  DoD.  A  trusting 
environment  where  there  is  a  need  and  the 
incentive  to  share  information  is  essential  to  the 
success  of  knowledge  management.  To  counter 
the  popular  saying  that  knowledge  is  power,  he 
quoted  ADM  Harold  Gehman.  "Knowledge, 
jealously  guarded  by  an  individual,  is  not  power. 
Quite  the  opposite,  it  detracts  from  the  power 
and  effectiveness  of  the  entire  organization." 
"We  must  adapt  to  new  information  flow. 
Everyone  must  “know”  what  everyone  else 
knows." 

Working  Group  Sessions 

The  workshop  attendees  were  organized  into 
four  subject  area  WGs  of  about  20  to  30 
members  each  plus  a  Synthesis  Group  to  address 
the  overall  goals  and  objectives. 

WG  1,  Management  of  Analytic  Knowledge, 
examined  the  manner  in  which  analytic 
knowledge  is  discovered,  expanded,  organized 
and  shared. 

WG  2,  Enabling  Collaboration,  examined  the 
process,  tools  and  techniques  that  support 
distributed  analytic  teams  in  performing  analytic 
tasks. 

WG  3,  New  Analysis  Challenges,  examined 
military  operations  analysis  questions  of  interest 
for  which  the  established  military  operations 


analysis  community  apparently  does  not  have 
answers. 

WG  4,  New  Analysis  Tools  and  Methods, 
examined  emerging  developments  in  analytical 
approaches  to  complex  systems  to  assess 
potential  applications  to  challenging  military 
operations  analysis  problems. 

The  Synthesis  Group  sought  to  discover  the 
common  themes  that  tie  together  the  efforts  of 
the  four  subject  area  WGs  and  provide  feedback 
to  the  groups  on  a  continuing  basis. 

In  the  following  sections,  the  efforts  and 
products  of  each  of  these  five  groups  will  be 
described  in  greater  detail,  sessions  will  be 
summarized  and  recommendations  will  be 
enumerated. 

Working  Group  1  —  Management  of  Analytic 
Knowledge 

Vision  Statement:  Informed  decisions  resulting 
from  effective  and  efficient  analyses  enabled  by 
the  sharing,  collaboration  and  creation  of 
knowledge. 

WG  1  examined  four  questions: 

•  How  is  analytic  knowledge  currently 
discovered,  captured,  expanded,  organized 
and  shared  within  the  DoD  analysis 
community? 

•  What  Knowledge  Management  techniques 
can  be  applied  to  improve  DoD  analysis? 

•  What  analytic  practices  (fast  turn-around, 
deliberate,  etc.)  would  make  use  of  analytic 
knowledge? 

•  What  recommendations  do  you  have  to 
improve  the  existing  analytic  practice  using 
knowledge  management? 

This  WG  examined  the  literature  that  is  defining 
the  field  of  Knowledge  Management.  The  group 
examined  the  manner  in  which  analytic 
knowledge  is  discovered,  expanded,  organized 
and  shared.  It  also  addressed  intelligent  search 
engine  technology  and  intelligent  agent 
technology  and  how  these  techniques  leverage 
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web-based  stored  data  to  meet  real-time  analysis 
requirements. 

WG  1  Session  Summary 

Session  1  -  Introduction  to  Topic;  The  Analytic 
Enterprise 

Session  2  -  Case  Study  on  the  Mobility 
Requirements  Study;  A  Portal  to  Analysis 
Knowledge 

Session  3  -  A  Conceptual  Framework  for  Using 
Knowledge  Management;  Case  Study 
Knowledge  Management  Lessons  from  the 
Synthetic  Theater  of  War 

Session  4  -  DoD  Practices  for  Acquiring  and 
Storing  Analytic  Knowledge 

Session  5  -  Working  Discussion 

Session  6  -  Working  Discussion 

Moving  From  “Siloed”  to  Enterprise 
Knowledge 

The  WG  identified  features  of  DoD 
organizations  involved  in  analysis  today, 
assessing  that  analytic  knowledge  tends  to  be 
protected  as  if  held  in  a  silo,  accessible  only  to 
the  organization  holding  the  knowledge. 


“Siloed”  Knowledge 

Enterprise  Knowledge 

Information  “silos” 
determined  by  organizational 
boundaries,  divisions, 
functions,  etc. 

Enterprise- wide  integration 
of  information  resources. 

Study  centered 

Issues  centered  - 
Information,  people,  tools, 
etc. 

Navigation  via  operating  and 
file  systems 

Navigation  via  hypertext 
browser  links  and  search 
engines. 

Reliant  on  internally 
generated  information. 

Integrates  external 
information  from  strategic 
partners,  suppliers, 
customers  and  third  party 
sources. 

Information  shared  via  e- 
mail. 

Information  shared  via 
knowledge  portals. 

Table  3.  Contrasting  Views  of  Knowledge  Access 


In  contrast  to  this  current  state  of  "Siloed” 
Knowledge,  the  WG  defined  corresponding 
features  of  the  desired  state  of  enterprise 


knowledge  where  organizations  can  openly 
access  knowledge  from  the  entire  analytic 
enterprise. 

Lessons  from  the  Private  Sector 

A  study  recently  conducted  by  the  Working 
Council  for  Chief  Information  Officers 
demonstrates  that  there  are  approaches  already 
implemented  in  industry  from  which  DoD  may 
benefit.  It  should  be  emphasized  that  these  are 
proven  technologies  and  techniques.  In  fact, 
some  of  these  technologies  and  techniques  are 
either  prototyped  or  in  place  in  DoD.  For 
example,  both  ASD(C3I)  and  US  Joint  Forces 
Command  have  implemented  policies, 
techniques  and  technologies  that  are  adapted 
from  industry.  These  include: 

Thesaural  Browsers  —  These  browsers  generate 
lists  of  related  terms  and  resources  allowing 
users  to  take  advantage  of  otherwise  unexpected 
associations  between  seemingly  unrelated 
information  sources  and  types. 

Personalized  Intranet  Portals  —  The  creation  of 
customizable  intranet  interfaces  allows  analysts 
selective  access  to  information  resources 
required  to  perform  their  jobs,  enables  analyst 
automation  of  simple,  routine  tasks  and  creates  a 
vehicle  for  a  central  information-management 
body  to  “push”  relevant  content  to  distributed 
users  in  real  time. 

Cultural  Change  —  Make  collaboration  and 
knowledge  sharing  a  component  of  annual 
performance  counseling  and  provide  incentives 
(awards  and  commendations)  for  initiatives  that 
improve  collaboration  and  knowledge  sharing. 

Capture  and  Reuse  Best  Practices  —  Best- 
practice  teams  in  analysis  divisions  or  analysis 
units  leverage  their  subject  expertise  to  discover 
and  codify  best  practices,  which  are  then  passed 
on  to  a  corporate  knowledge-management  group 
that  helps  propagate  the  use  of  best  practices 
across  the  enterprise 

Information  Expiration  Mechanisms  —  Intranet 
content  is  automatically  identified  by  date  of 
creation  and/or  last  modification;  dated 
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documents  are  flagged  and  routed  to  their 
authors,  who  review  them  for  relevancy  and 
decide  to  archive,  delete,  modify  or  reinsert  the 
documents  back  into  the  corporate  database. 

Content  Validation  Protocols  —  A  process  must 
be  put  in  place  to  prevent  the  propagation  of 
inconsistent  or  incorrect  information  throughout 
the  enterprise. 

A  Strategy  for  Improving  Operations 
Analysis  in  DoD 

The  strategy  recommended  by  the  WG  to 
improve  the  practice  of  military  operations 
analysis  in  DoD  through  the  application  of 
enterprise  knowledge  management  is: 

•  The  Joint  Staff,  J-8,  and  OASD  (C3I), 
provide  leadership  in  the  development  and 
implementation  of  policies  that  both  enable 
and  foster  knowledge  sharing  and  then 
follow  through  with  the  facilitation  of 
change  to  realize  knowledge  sharing. 

•  OASD  (C3I)  take  the  lead  to  inform  DoD 
leaders  of  the  value  to  be  derived  from 
knowledge  sharing  in  order  to  change  the 
mindset  of  the  leaders.  Collaboration  and 
knowledge  sharing  must  be  promoted  within 
the  leadership  and  the  community  to  change 
the  culture  from  protecting  knowledge  to 
sharing  knowledge. 

•  DoD  organizations  and  agencies  must 
overcome  the  barriers  to  knowledge 
management  implementation. 

•  OASD  (C3I)  develop  policies  and 
procedures  that  result  in  continuing 
education  and  training  in  knowledge 
management  practices  and  technologies. 

The  DoD  Sponsors  should  prototype 
collaboration  and  knowledge  sharing  processes 
for  specific  military  operations  analysis 
problems  that  may  benefit  from  these 
approaches.  These  efforts  should  be  used  to 
illustrate,  from  the  bottom  up,  the  value  of 


collaboration  and  knowledge  sharing  in  evolving 
the  practice  of  analysis. 

Description  and  Discussion  of  WG  1  Findings 
and  Recommendations 

There  are  many  obstacles  to  collaboration  and 
knowledge  sharing  in  the  pursuit  of  military 
operations  analysis  in  DoD.  Policies  that  are 
currently  in  place  make  it  impossible  for  us  to 
use  existing  knowledge  tools.  The  lack  of  a 
single  coherent  GIG  or  infrastructure  precludes 
both  technique  and  technology  solutions  that  are 
tested  and  available.  The  pace  of  technological 
change  is  daunting;  we  need  a  plan  and  a 
framework  for  rational  decision  making  that 
allows  change  to  take  advantage  of  marketplace 
innovation  while  mitigating  the  impact  of 
change  on  organizations  and  their  budgets.  The 
analytic  community  must  find  the  social  or 
economic  model  that  provides  the  incentives  to 
individuals  to  share  information.  At  its  most 
fundamental  level,  collaboration  and  knowledge 
sharing  will  only  be  a  success  if  it  is  obvious  to 
the  individuals  involved  that  it  is  in  their  interest 
to  participate.  Organization  and  individual 
resistance  to  required  change  has  to  be 
overcome.  There  are  individual  and 
organizational  incentives  in  place  now  that  cause 
individuals  to  hoard  information.  Individuals 
use  the  information  they  have  as  a  means  to 
cement  their  importance.  This  is  probably  the 
single  biggest  hurdle  that  must  be  overcome. 
Change  is  hard;  there  must  be  something  in  it  for 
each  individual  analyst  to  embrace  the  concepts 
for  collaboration  and  knowledge  sharing. 

There  is  a  need  to  standardize  the  practice  of 
Military  Operations  Analysis,  that  captures  those 
activities  and  procedures  that  are  common  to 
most  analyses.  Standard  analytic  practices 
facilitate  collaboration  and  knowledge  sharing 
by  reusing  proven  methods  and  analytic  product 
frameworks.  Standard  practices  that  may  be 
accessed  through  the  intranet  or  Internet  also 
help  resolve  gaps  in  experience  levels  that  may 
exist  in  an  organization's  analytic  capability. 

The  need  for  security  and  the  policies  that 
enforce  security  are  important,  however, 
information  security  is  probably  the  second 
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biggest  (after  individual  resistance)  stumbling 
block  to  improved  analytical  practice.  We  must 
make  a  very  careful  assessment  of  both  the  need 
for  highly  compartmentalized  security  practices 
(need-to-know)  and  the  way  in  which  these 
policies  are  implemented.  The  SECRET 
Internet  Protocol  Router  Network  (SIPRNET)  is 
already  challenging  many  of  the  old  shibboleths. 
This  is  not  to  say  we  should  be  careless,  but 
rather  we  must  assess  the  payoff  of  greater 
sharing  versus  the  risks  associated  with  sharing. 
The  rewards  for  sharing  are  lowered  cost  in 
terms  of  the  time  it  takes  to  perform  analysis.  If 
speed  of  analysis  is  as  important  as  our 
leadership  insists  it  is,  then  barriers  must  come 
down. 

WG  1  had  thirteen  specific  recommendations 
arrayed  across  categories  of  Strategy,  People, 
Process,  and  Technology.  They  are: 

Strategy 

•  The  DoD  Sponsor  should  implement 
knowledge  management  techniques, 
particularly  to  establish  an  intranet  portal,  in 
support  of  an  important  study  with  suitable 
scope,  schedule  and  content  that  partners 
DoD  components  MORS,  industry  and 
academia.  The  elements  that  should  be 
considered  along  with  strategy  include 
people,  process  and  technology. 

People 

•  DoD  and  MORS  lead  an  assessment  of  the 
capability  to  provide  a  baseline  inventory  of 
analysts  for  DoD  operations  research  studies 

•  MORS  assess  its  capability  to  establish  a 
Knowledge  Management  Professional 
Development  Program  for  the  DoD  Analytic 
Enterprise  that  would  include  the 
identification  of  skills  and  products,  the 
refinement  of  skill  levels,  and  the  training  of 
analysts. 

•  The  DoD  Sponsors  evaluate  successful  skill- 
inventory  systems  (e.g.  Northrop-Grumman 
and  Y2K)  for  implementation  within  the 
DoD  Analytic  Enterprise. 


Process 

•  J-8  accept  leadership  with  OASD  (C3I)  of 

an  evolving  analysis  process  for  the  purpose 
of  improving  analysis  in  a  future  analysis 
state.  Notionally,  the  evolving  process 
could  be  closely  interfaced  with  an  evolving 
knowledge  management  process  as  shown  in 
Figure  2. 


Figure  2.  Interrelationships  Related  to 
Collaboration,  Creation  and  Reuse 

•  DoD  determine  the  business  rules  for 
contributing  analytic  knowledge.  For 
example,  what  are  the  current  rules  that 
control  the  access  to  past  studies  and  how 
could  these  rules  be  changed  to  improve 
access  such  as  those  stored  at  FFRDCs?  A 
special  case  is  the  identification  of  the 
business  rules  for  administering  and 
controlling  an  analyst’s  portal. 

•  The  DoD  Sponsors,  with  the  assistance  of 
MORS,  identify  new  types  of  information 
required  for  future  analyses  (e.g.  foreign 
demographics,  information  warfare  related, 
economic  information,  national 
infrastructure  related,  chemical/biological, 
etc.) 

•  The  DoD  Sponsors  identify  current  centers 
of  expertise  required  for  improved  analyses. 

•  The  DoD  Sponsors  take  steps  to  systematize 
the  organization  of  information  by  using 
principles  from  traditional  library  and 
information  sciences  in  the  electronic  realm. 

•  The  DoD  Sponsors,  with  the  assistance  of 
OASD  (C3I),  develop  and  use  an 
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information  architecture  that  structures  the 
design  of  information  organization,  labeling, 
navigation  and  indexing  systems  to  support 
both  browsing  and  searching. 

•  The  DoD  Sponsors  identify  and  use  a  set  of 
metrics  for  the  business  case  preparation  and 
performance  evaluation  of  the  Analyst’s 
Portal. 

•  The  DoD  Sponsors  create  knowledge¬ 
sharing  incentives  such  as:  Performance 
Reviews  that  emphasize  collaboration  and 
knowledge  sharing  and  recognition  for 
individuals  show  collaborate,  share 
knowledge  and  otherwise  support  the 
organizations  efforts  in  this  area. 

Technology 

•  DoD  concentrate  on  applying  Commercial- 
Off-The-Shelf  (COTS)  tools  as  a  key  critical 
success  factor  for  building  analytical 
Knowledge  Management  portal.  Example 
technologies  include: 

■  Thesaural  Browsers 

■  Personalized  Intranet  Portals 

■  Federated  Knowledge  Management 

■  Information-Expiration  Mechanisms 

Working  Group  2  —  Enabling  Collaboration 

Vision  Statement:  Collaborative  analysis  is  an 

analytic  activity  that  involves  the  sharing  of 

data,  tools  and  expertise  across  organizational 

boundaries  to  achieve  a  common  goal. 

WG  2  examined  four  questions: 

•  How  is  collaboration  currently  used  in  the 
DoD  analysis  community? 

•  What  formal  and  informal  collaboration 
techniques  can  be  applied  to  improve  DoD 
analysis? 

•  What  analytic  practices  (local  study, 
centrally  managed  study,  etc.)  would  make 
use  of  collaboration  support? 

•  What  recommendations  do  you  have  to 
improve  the  existing  analytic  practice  using 
collaboration? 


The  working  group  participants  determined  that 
the  first  and  third  questions  were  very  similar 
and  combined  their  consideration. 

WG  2  Session  Summary 

Session  1  -  Current  State  of  Collaborative 
Analysis 

Session  2  -  When,  Where  and  What  Kind  of 
Collaborative  Analysis  is  Desired? 

Session  3  -  Collaborative  Analysis  Policy  Issues 

Session  4  -  Organizing  for  Collaborative 
Analysis 

Session  5  -  Emerging  Techniques  that  Enable 
Collaborative  Analysis 

Session  6  -  Locating  Analysis  and  Subject 
Matter  Expertise 

Where  Does  Collaboration  Exist  in  Analysis 
Today? 

There  are  three  principal  types  of  analyses  in 
which  collaboration  is  a  reality  today. 

Analysis  in  the  unified  commands  is 
collaborative  by  nature  due  to  the  military, 
political,  inter-agency  and  international 
considerations.  In  this  real-time  environment, 
the  decision-cycle  is  so  compressed  that  analysis 
must  be  agile  to  be  useful.  If  tools,  data  and 
talent  are  not  available  when  called,  the  decision 
will  be  made  without  analytical  insight. 

Analysts  in  US  European  Command  (EUCOM) 
have  learned,  for  example,  that  they  are 
frequently  asked  to  provide  deployment  and 
logistical  analysis.  Collaborative  work  is  made 
more  difficult  by  geographic  separation  and 
Operations  Security  (OPSEC)  concerns. 

Within  the  Beltway,  collaborative  analysis  is 
frequently  directed  to  answer  Congressional 
Language  or  to  support  the  Planning, 
Programming  and  Budgeting  System  (PPBS) 
decisions.  In  this  environment,  joint  and  service 
considerations  pre-dominate.  Unlike  the 
environment  previously  described,  these 
activities  are  important,  but  not  so  urgent  that 
tools,  data  and  skills  cannot  be  massed  and 
employed  in  a  deliberate,  formal  way.  A 
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distinguishing  characteristic  of  this  kind  of 
collaborative  work  in  a  hierarchical  oversight 
structure  that  guides  the  analysis  from  cradle  to 
grave. 

The  third  collaborative  analysis  type  is  the 
service/agency  study  performed  to  meet  an 
internal  requirement.  Intra-service  and  inter¬ 
service  collaboration  is  driven  by  the  needs  of 
the  service  customer.  Analyses  of  Alternatives 
(AoAs),  for  example,  for  Acquisition  Category  I 
(ACAT-I)  systems  frequently  require  a  joint 
context.  Consultation  and  rendered  assistance 
across  organizational  boundaries  may  be  formal 
(by  memorandum)  or  informal  (by  professional 
courtesy). 

Formal  and  Informal  Techniques  that  Enable 
Collaboration 

Collaboration  can  be  encouraged  and  assisted  by 
both  formal  and  informal  mechanisms.  One 
motivation  for  an  organization  to  collaborate  is 
to  acquire  the  use  of  tools  and  expertise  to  meet 
infrequent  requirements  (e.g.,  support  to  Joint 
exercises  or  contingency  plan  development). 
EUCOM  and  the  Center  for  Army  Analysis 
(CAA)  believe  that  this  arrangement  works  best 
when  working  relationships  have  been 
established  in  advance  and  are  rehearsed.  In  this 
way,  the  supporting  organization  understands 
the  decision  making  processes  of  the  supported 
organization  and  the  hardware/software/security 
requirements  for  interoperability.  While 
collaborative  work  can  be  done  from  a  distance, 
EUCOM  prefers  on-site  support.  Co-location 
helps  allay  fears  concerning  operations  security, 
responsiveness  and  miscommunication. 

The  existence  of  an  “honest  broker”  for  data 
encourages  participants  to  share  their  data 
holdings.  The  Joint  Data  Support  (JDS)  Office 
has  earned  the  trust  of  the  Services,  the  Joint 
Staff  and  other  offices  in  the  Secretariat,  by  the 
way,  it  has  controlled  the  data  entrusted  to  it 
during  Mobility  Requirements  Study  2005 
(MRS -05).  Perhaps  more  important  than  its  role 
as  a  “gatekeeper,”  is  the  service  JDS  provides  to 
the  community  by  cross-checking  the  data  it 
receives  against  other  authoritative  sources  to 
identify  data  anomalies.  These  anomalies  are 


provided  back  to  the  source  for  resolution, 
improving  the  veracity  and  consistency  of  DoD 
data  holdings. 

Informal  trust-building  activities  like  MORS  and 
positive  experiences  with  collaborative  analysis 
help  to  overcome  organizational  and  individual 
resistance  to  teaming. 

It  is  important  to  remember  that  collaborative 
analysis  is  not  an  appropriate  technique  for  all 
situations.  A  quote  attributed  to  Napoleon  is  “I 
would  rather  fight  allies  than  be  one.”  Sharing 
common  ends  does  not  guarantee  agreement  on 
the  ways  and  means  to  be  used  to  get  there. 
Group  pressure  to  cooperate  can  result  in 
compromises  that  preserve  the  status  quo  when 
what  was  really  needed  was  creativity  and  a  bold 
shift. 

LTC  Jerry  Glasow  presented  a  paper  in  the  WG 
that  classifies  collaborative  methods  as  being 
either  “input-based”  or  “output-based.”  MRS- 
05  typifies  the  former  while  the  Phased  Threat 
Distribution  (PTD)  Study  is  an  example  of  the 
later.  The  “input-based”  approach  reasons  that 
if  the  participants  accredit  (1)  the  data,  (2)  the 
tools  and  (3)  the  analysts,  the  results  will,  in 
turn,  be  accredited.  When  this  approach  is 
implemented  by  a  sequence  of  models  (e.g.,  air 
defense  precedes  deployment  precedes  force-on- 
force  evaluation)  a  single  point  failure  can  delay 
the  entire  project.  It  also  tends  to  produce  point 
estimates  rather  than  a  range  of  possible 
outcomes  for  consideration. 

Two  advantages  of  the  “output-based”  approach 
are  parallel,  independent  analysis  and  multiple 
results  that  can  be  compared.  The  most  difficult 
part  of  this  approach  can  be  determining  why  the 
results  are  different.  If  this  approach  is  used, 
several  integrated  teams  should  be 
commissioned  by  the  participants  rather  than 
putting  the  participants  in  competition  with  one 
another. 
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Data  Collection  and  Production 

Data  collection,  production,  preservation  and 
maintenance  were  recurring  themes  in  the  WG 
deliberations.  Jim  Methered,  citing  his 
experience  during  and  after  OPERATION 
ALLIED  FORCE,  is  vehement  that  data 
collection  must  be  a  component  of  the  Joint 
Task  Force  (JTF)  information  management  plan 
instead  of  the  “ad  hocery”  that  exists  today: 
“...scavengers  picking  over  small,  rapidly 
decaying,  incomplete  and  questionable  data  and 
then  fiercely  defending  the  scraps  gathered  from 
others.” 

A  potential  major  future  role  for  operations 
analysis  may  be  to  study  how  information  is 
used  and  develop  ways  of  managing  and  using  it 
better.  Process  engineering  in  peace  might 
facilitate  analysis  in  war  as  well  as  improve 
actual  and  simulated  decision  making  processes. 

Description  and  Discussion  of  WG  2  Findings 
and  Recommendations 

The  working  group  recommendations  are 
collected  under  three  headings,  people,  process 
and  products. 

People 

We  must  find  ways  to  identify  (and  evaluate) 
subject  matter  experts  to  make  access  to  such 
people  fast  and  to  provide  confidence  to  the 
analyst  that  the  SME  they  are  in  contact  with  is 
respected  in  the  subject  matter  domain  with 
which  they  are  associated. 

We  must  find  a  means  to  develop  operational 
experience  in  Civil  Service  1515s  and  uniformed 
analysts  who  do  not  hold  warfighting  specialties. 
The  alternative  is  to  find  a  way  to  integrate  non¬ 
analyst  operators  in  analysis  efforts. 

Process 

There  is  a  spectrum  of  collaborative  analysis 
processes,  and  each  has  its  strengths  and 
weaknesses  (bold  shifts  versus  consensus). 

•  We  can  organize  to  perform  competitive 
analyses  to  find  new  ideas. 


•  We  can  organize  to  search  for  a  “common 
ground”  solution.  Collaborative  analysis  is 
a  “team  sport”  and  requires  practice. 

Operational  Analysis  support  to  Commanders  in 
the  field  is  most  effective  when  a  mindset  of 
“Train  as  you  Fight”  is  embedded  in  training  and 
analysis  practice.  Ways  must  be  found  to  re¬ 
establish  formal  liaison/exchanges  to  improve 
the  quality  of  analysis. 

Today,  collaborative  analysis  takes  longer  to 
perform  than  analysis  done  in  isolation.  In  the 
future,  timelines  can  be  shortened  if 
informational  exchanges  and  liaison  between 
analytic  organizations/agencies  are 
institutionalized  in  our  business  practice. 

In  the  short-term,  it’s  important  that  we 
reconcile  our  demands  for  data  with  our  ability 
to  produce  it.  Dr  Chris  Lamb,  ASD(S&TR),  is 
currently  leading  an  effort  with  the  services  and 
the  intelligence  agencies  to  better  coordinate  our 
efforts.  It  also  makes  good  sense  to  plan  in 
advance  for  the  long-lead  data  that  are  common 
to  many  of  the  studies  we  do  (e.g.,  Time-phased 
Force  Deployment  Data  (TPFDDs)).  In  the 
long-term,  data  collection  in  the  operational 
environment  needs  to  evolve  from  something 
that  interferes  with  task  performance  to 
something  that  is  transparent  to  people 
performing  their  day-to-day  tasks.  Similarly, 
directives  to  perform  large,  collaborative 
analysis  needs  to  be  harmonized  with  the  day-to- 
day  analytic  workload. 

•  Support  independent,  accredited  sources  for 
data  to  speed  collaborative  effort. 

•  Find  ways  to  identify  data  to  source  new 
applications  and  phenomena. 

•  Data  collection  must  be  deliberate  and  done 
in  such  a  way  that  it  does  not  require 
additional  effort  by  operators. 

•  Synchronize  data  production  with  data 
demands  and  institutionalize  production  and 
update  of  long-lead  items  (e.g.,  extend  the 
Joint  Munitions  Effectiveness  Manual 
(JMEM)  to  include  developmental  systems). 
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•  Harmonize  schedule  of  major  collaborative 
projects  and  service  workload. 

Products 

Compressed  decision-cycles  and  the  tendency 
toward  using  briefings  rather  than  reports  as 
deliverables  has  lessened  our  use  of  literature 
search  and  our  discipline  to  thoroughly 
document  what  we  do. 

We  need  improvements  in  literature  search 
capabilities,  identifying  work  in  progress  that 
may  affect  the  project  at  hand. 

We  must  improve  our  study  documentation 
techniques,  allowing  for  reproducibility  of 
results  and  reuse  of  data  in  addition  to 
descriptions  of  our  procedures  and  findings. 

Working  Group  3  —  New  Analysis  Challenges 

WG  3  examined  a  single  focus  question: 

•  What  new  military  operations  problems 
should  the  analysis  community  consider  in 
its  evolution  of  analytic  practice?  Or  stated 
differently,  what  problems  are  not  well 
served  by  the  current  state  of  the  practice? 

The  working  group  examined  military 
operations  analysis  questions  for  which  the 
established  military  operations  analysis 
community  apparently  does  not  have  the 
answers.  The  working  group  was  about  new 
problems,  problems  not  involving  conventional 
warfare,  and/or  post-Cold  War  problems. 
Actually  there  are  old  problems  that  belong  here 
as  well.  The  issue  isn’t  whether  a  problem  is 
new  or  old,  but  rather  whether  the  military 
operations  research  community  has  succeeded  in 
addressing  it  satisfactorily.  An  example  is  the 
problem  of  determining  the  value  of  a  common 
operational  picture  to  the  warfighter. 
Presentations  to  this  working  group  came  from 
the  major  analysis  players  such  as  the  Services’ 
Studies  and  Analysis  shops,  from  warfighters 
like  the  US  Joint  Forces  Command  (JFCOM) 
and  US  Pacific  Command  (PACOM),  and  from 
decision  makers  like  OSD,  the  Joint  Staff  and 
the  Services'  warfare  requirements  and 
assessment  organizations. 


WG  3  Session  Summary 

Session  1  -  New  Analysis  Challenges 

Session  2  -  Challenges  in  Army  Modeling 

Session  3  -  Analysis  Challenges  for  Joint 
Operations  -  A  USCINCPAC  Perspective 

Session  4  -  New  Analysis  Challenges  in  Navy 
Programs  and  Fleet  Battle  Experiments 

Session  5  -  New  Analysis  Challenges  in  Air 
Force  Programs  and  Logistics  Analysis 
Challenges 

Session  6  -  Analytical  Challenges  of  the  USMC 

Analysis  Problems  that  are  not  Well-Served 
by  Current  Analysis  Techniques  or 
Approaches 

With  the  end  of  the  Cold  War,  the  Department 
of  Defense  and  thus  the  DoD  analytical 
community,  are  faced  with  the  demand  of 
conducting  and  thus  analyzing  a  variety  of  new 
operations.  These  new  operations  require  new 
methods  of  analysis  for  planning,  conceiving 
concepts  of  operations,  defining  doctrine,  and 
analyzing  effectiveness.  Some  examples  of 
these  new  domains  are: 

•  High  Intensity/Low  Casualty  Conflict. 

•  Military  Operations  Other  Than  War 
(MOOTW). 

•  Humanitarian  Operations. 

•  Peacetime  Engagement. 

•  Operations  in  Urban  Terrain. 

•  Technology  Insertion  (Digital  Battlefield, 
Tomorrow’s  Dismounted  Warrior). 

•  Chemical/Biological  Warfare  (Including 
Contagion). 

•  Asymmetric  Threats  (domestic  terrorism, 
political  manipulation,  homeland  defense). 

•  Special  Operations. 

•  Information  Operations/Warfare. 
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Problems  with  the  Current  Emphasis 

The  current  emphasis  on  modeling  and 
simulation  with  its  resulting  de-emphasis  on 
other  analytical  tools  and  methods  is  limiting  the 
analysis  toolkit.  With  the  amount  of  time  and 
resources  required  for  scenario  development  and 
data  collection  to  run  the  current  models  and 
simulations,  some  other  alternatives  are  clearly 
needed.  For  example,  methods  of  scoping 
analysis  that  can  quickly  look  at  the  alternatives 
and  potentially  set  the  stage  for  further,  larger, 
study  are  needed.  These  scoping  tools  can  serve 
to  pare  down  the  alternatives  and  allow  the  focus 
on  the  important  question  to  evolve.  In  addition, 
simpler  analysis  tools  such  as  spreadsheets 
should  be  used  and  exchanged  within  the 
analytical  community  as  a  rapid  means  of 
knowledge  transfer.  Finally  the  analyst  should 
be  taught  to  combine  the  results  from  all 
pertinent  tools  and  analytical  methods  to  more 
thoroughly,  accurately  and  quickly  answer  the 
question  at  hand. 

With  the  increasing  complexity  of  today’s 
military  scenarios  and  high  technology  weapons, 
there  is  a  greater  need  for  the  analysis 
community  to  be  able  to  identify  and 
communicate  the  effects  of  the  factors  and  their 
interrelationships.  It  is  especially  important  to 
be  able  to  conduct  and  communicate  the  trades 
conducted  between  factors  that  typically  would 
not  be  linked.  For  example,  an  aircraft  carrier 
can  be  the  platform  used  for  Joint  Strike  Fighter 
(JSF)  launches,  and  a  typical  analysis  may  look 
at  the  number  of  targets  put  at  risk  by  various 
sizes  of  JSF  fleets.  However,  an  interrelated 
measure  that  should  be  considered  is  the  deck 
space  that  is  being  taken  up  by  the  JSF  fleet,  that 
may  potentially  be  used  for  such  things  as 
helicopter  medical  evacuation  (MEDEVAC) 
support,  two  very  important  missions  with 
interrelations  that  must  be  considered. 

There  are  many  factors  that  effect  the  outcome 
of  a  battle,  but  do  not  have  directly  measurable 
combat  effects.  These  factors  must  be  taken  into 
account,  and  their  effects  must  be  traced  to 
operational  and  strategic  outcomes.  These 


factors  include  such  things  as: 

•  Logistics. 

•  Bandwidth  requirements. 

•  Engagement/presence  on  PERSTEMPO  and 
OPSTEMPO. 

•  Infrastructure  investments. 

•  Readiness  postures. 

•  Various  force  structure  alternatives. 

The  behavior  of  individuals,  small  units  and 
adversary  leaders  must  be  captured  in  our 
evolving  analytical  tools.  The  effect  and 
measurement  of  behavior  factors  at  the 
individual  or  small  unit  levels  are  important 
considerations,  especially  in  the  analysis  and 
evaluation  of  combat  systems.  For  example,  the 
increase  in  situational  awareness  for  the 
individual  foot  soldier  is  probably  a  good  thing 
to  a  point,  until  information  overload  occurs  or  if 
the  equipment  used  to  give  the  situational 
awareness  decreases  the  individual’s 
effectiveness  in  other  areas,  such  as  mobility. 

On  the  other  hand,  being  able  to  quantify  the 
characteristics  and  motivations  of  an  adversary 
leader  would  allow  the  identification  of  the 
leader’s  or  country’s  centers  of  gravity  and  may 
assist  the  planning  process  in  identifying 
effective  target  sets  that  would  influence  the 
enemy’s  course  of  action. 

Methods  for  quantifying  risks  of  all  sorts  must 
be  developed  and  institutionalized. 

The  frequency  of  calls  for  quick-tum  around 
analysis  is  increasing.  These  analyses  take 
many  forms,  such  as  support  for  testimony  to 
Congress  or  support  to  ongoing  warfighting 
efforts.  To  answer  these  types  of  questions,  the 
analyst  needs  a  set  of  tools  that  support  these 
quick  analysis  activities.  These  tools  need  to 
capture  the  important  elements  that  would  be 
drawn  out  by  more  thorough  analysis  but  run 
quickly  enough  to  support  answering  the 
question  when  the  answer  will  still  be  desired 
and  relevant. 
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Description  and  Discussion  of  WG  3  Findings 

and  Recommendations 

There  were  three  broad  recommendations  that 

were  defined  by  WG  3. 

•  As  the  analytical  community  evolves,  it 
needs  to  find  tools  to  deal  with  new  forms  of 
operations,  find  methods  to  trace  known 
factors  and  interactions  to  their  logical 
combat  output,  and  find  tools  that  will 
provide  accurate  and  thorough  analysis  in  a 
short  timeline. 

•  The  DoD  analytical  community  needs  to 
nurture  analysts  and  emphasize  analytical 
thinking  rather  than  focusing  on  simulation 
and  modeling  development.  The  community 
must  project  a  more  balanced  view  of  model 
capabilities  as  well  as  limitations,  and 
reverse  the  recent  trend  toward  blind  faith 
toward  large  models  as  the  solution  to  all 
analytical  problems. 

•  The  analytical  community  needs  to  find 
means  for  characterizing  and 
communicating  the  impacts  of  uncertainty. 
For  example  risk,  such  as  decision  risk 
based  on  the  fact  that  various  factors  may 
not  have  been  considered  in  the  current 
analysis,  need  to  be  characterized  and 
communicated  to  the  decision  maker. 

Working  Group  4  —  New  Analysis  Tools  and 

Methods 

WG  4  examined  four  questions: 

•  What  currently  emerging  developments  in 
analytical  approaches  to  complex  systems 
can  be  applied  to  challenging  military 
operations  analysis  problems?  Consider 
areas  such  as  chaos,  complexity  theory,  etc. 

•  To  what  DoD  analyses,  particularly  the  new 
analytic  challenges  defined  by  WG  3,  can 
these  approaches  be  applied? 

•  What  recommendations  do  you  have  to 
improve  the  existing  analytic  practice  in  the 
near  term  by  using  new  analysis  tools  and 
methods? 


•  What  recommendations  do  you  have  for 
areas  of  research  for  the  next  generation  of 
military  operations  analysis? 

The  New  Analysis  Tools  Working  Group  sought 
to  identify  emerging  tools  and  techniques  for 
conducting  operations  analysis,  given  the  new 
challenges  that  face  the  military  operations 
research  community  in  the  21st  century. 

To  accomplish  this,  MORS  leadership 
assembled  representatives  from  domains  that  are 
not  normally  part  of  the  traditional  MORS 
population.  Experts  with  expertise  in 
Command,  Control,  Communications, 

Computers  and  Intelligence  (C4I)  systems, 
intelligence,  policy  analysis,  artificial 
intelligence  and  knowledge  management  were 
present.  This  eclectic  set  of  skills  resulted  in 
information  exchange  that  exceeded 
expectations,  a  vision  focused  on  providing  a 
foundation  of  consistent  and  compatible  data,  as 
well  as  an  architecture  that  could  provide 
connectivity  between  command  and  control 
systems  and  analytic  modeling  environments. 

The  kinds  of  connectivity  and  consistency  that 
are  envisioned  in  this  product  are  believed  to  be 
absolutely  essential  for  providing  commanders 
and  staffs  in  the  field  with  flexible  automated 
planning  support,  and  keeping  analysis  relevant 
in  the  highly  uncertain  future  that  the  DoD  faces. 

Perhaps  most  importantly,  it  was  concluded  that 
for  the  most  part  there  is  no  technological 
impediment  to  achieving  the  vision  that  is 
articulated  herein,  and  that  these  kinds  of  efforts 
are  an  essential  part  of  achieving  the  knowledge 
management  “culture”  that  has  been 
implemented  successfully  in  many  commercial 
enterprises. 

WG  4  Session  Summary 

Session  1  -  Mapping  “New  Tools”  to  “New 
Needs” 

Session  2  -  Joint  Warfare  System,  Joint 
Modeling  and  Simulation  System,  Land 
Formation  Model 


15 


Session  3  -  Army  Modeling  and  Simulation 
Office,  Army  Artificial  Intelligence  Center,  Joint 
Data  Support 

Session  4  -  C4I  Model,  Project  Albert  and 
Operational  Synthesis  and  DTI 

Session  5  -  Mapit  and  Dynarank 

Session  6  -  IWEDA  and  Terrain  Modeling 

Assumptions  and  Scope 

Because  the  solution  space  within  which  the 
working  group  was  operating  the  group  made 
initial  assumptions  to  bound  its  discussions.  In 
some  cases  the  assumptions  were  found  to  be 
too  restrictive  and  were  later  relaxed. 

The  first  assumption  is  that  Knowledge 
Management  experts  will  create  a  repository 
infrastructure,  and  that  infrastructure  will  be 
available  for  use  to  store  and  distribute  analysis 
domain  knowledge. 

Analysts  within  DoD  have  been  modeling  and 
simulating  for  a  long  time.  There  is  an  existing 
set  of  recommendations  in  the  DoD  Modeling 
and  Simulation  Master  Plan.  These 
recommendations  are  assumed  to  be  good,  and 
future  recommendations  should  deviate  only 
when  needed. 

There  are  architectures  available  to  us  to  move 
information  that  would  be  in  a  knowledge 
management  oriented  system  and  culture. 
Discussion  with  experts  from  the  C4I  domain 
revealed  that  the  assumption  does  not  hold  with 
respect  to  the  Defense  Information  Infrastructure 
Common  Operating  Environment  (DII  COE) 
standards. 

The  Analysis  Functional  Working  Group  of  the 
DoD  Executive  Committee  for  Modeling  and 
Simulation  (EXCMS)  has  a  valid  list  of  analysis 
deficiencies  that  needs  to  also  be  considered. 

The  WG  focused  on  how  to  do  big  studies  better 
and  then  began  to  look  at  minute-long  decision 
cycles  (from  WG  3)  to  balance  out  the 
discussions. 


Analytical  Requirements 

Although  warfare  doctrine  and  military  missions 
are  evolving,  there  are  constant  /  recurring 
analytic  requirements  that  can  and  should  be 
planned  for: 

•  Force  assessment. 

•  Planning  and  execution. 

•  Deliberate  planning. 

•  Crisis  action  planning. 

•  System  effectiveness  and  trade  off  analysis. 

•  Concept  and  doctrine  development  and 
assessment. 

These  missions  are  fairly  constant  in  the 
workspace  of  military  operations  research.  There 
is  ever  reason  to  believe  they  will  continue  to  be 
the  main  missions  of  the  future.  They  form  the 
space  around  which  to  structure  the  new  tools 
for  analysis  challenges. 

Because  the  group  began  deliberation  at  the 
same  time  WG  3  was  meeting,  a  generic  set  of 
decisions  that  could  be  supported  by  analysis 
was  used  to  stimulate  discussion.  Over  the 
course  of  the  workshop,  the  group  considered 
WG  3  inputs  as  they  arrived. 

One  notable  possibility  that  was  added  by  the 
C4I  domain  was  the  existence  of  decision  aids 
that  are  either  currently  embedded,  or  planned 
for  in  near  real-time  command  and  control 
systems.  These  systems  would  benefit  from 
analytically  sound  underlying  algorithms  and  the 
group  felt  strongly  that  the  analysis  community 
would  need  to  be  an  active  part  of  their 
development. 

Accomplishing  these  analytic  missions  requires 
descriptive  tools  (models  and  simulations), 
prescriptive  tools  (choice  and  resource 
allocation),  reliable  and  consistent  data. 

A  significant  body  of  work  is  underway  in  the 
DoD  that  advances  the  state-of-the-art  in 
modeling  “infrastructure.”  This  work  seeks  to 
correct  deficiencies  inside  mission  areas. 
However,  it  fails  to  capitalize  on  related  work 
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(schedule  and  resource  constraints).  This  results 
in  inefficient  and  inconsistent  modeling  and 
analysis  efforts. 


A  Knowledge  Management  Framework  for 
Analysis 


During  the  conference,  we  saw  the  knowledge 
management  framework  (the  dark  gray  or  blue 
boxes)  as  a  hierarchy  of  knowledge  products 
built  on  the  base  of  known  facts  and  derived 
meanings  as  shown  in  Figure  3. 
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Figure  3.  Analysis  Process  in  the  Context  of 
Knowledge  Management 

One  of  the  insights  from  our  work  is  that  the 
application  of  knowledge  management  within 
the  analysis  community  is  actually  a  process  of 
becoming  conscious  of  the  importance  of 
building  and  sharing  the  knowledge  of  our 
analysis  enterprise  —  both  subject  area  and 
analysis  expertise. 

The  analysis  process  (shown  in  light  gray  or 
yellow)  produces  knowledge  products.  The 
essence  of  analysis  has  always  been  to  create 
knowledge  from  the  raw  materials  of  facts  and 
meanings. 

One  aspect  of  wisdom  —  the  highest  level 
knowledge  product  shown  in  the  original 
framework  (and  missing  from  the  far  right  side 
of  this  diagram)  —  results  from  continuing  to 
develop  and  use  analysis  capabilities  over  time. 


Figure  4  provides  a  slightly  different  view  of  the 
analysis  process  and  products  from  the  previous 
slide.  It  highlights  the  product  areas  that  the 
working  group  might  consider  as  areas  for 
improvements. 
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Figure  4.  Analysis  Process  Alternative  View 

Much  of  the  discussion  centered  on  tools  and 
data,  particularly  the  need  to  integrate 
information  generated  from  multiple  tools, 
sequence  data  flows  from  one  tool  to  another, 
and  to  ensure  that  the  data  objects  were 
consistent  such  that  they  could  be  used  as  is  or 
logically  transformed  to  meet  modeling 
requirements.  The  term  “model,”  defined  in  the 
broadest  sense  as  the  abstract  representation  of 
any  problem,  was  frequently  discussed  as  a  key 
tool. 

Another  emphasis  area  was  to  reduce  the  data- 
to-tool-to-information  cycle  time  sufficiently  to 
meet  the  needs  of  the  customers  with  very  short 
decision  timelines.  Traditionally,  these 
customers  have  relied  on  back-of-the-envelope 
estimates,  “quick  and  dirty”  analyses,  to  offer 
insights  to  support  decisions.  Since  these 
decisions  are  frequently  high-level  and  wide 
ranging,  the  analysis  community  would  like  to 
design  tool  and  data  solutions  that  offer  both  a 
high  quality  and  timely  response. 

Decision  support  needs  centered  on  visualization 
tools  —  the  ability  to  clearly  represent  complex 
decision  spaces,  dissimilar  tradeoffs  and  the  like 
—  that  have  not  yet  been  identified.  The  group 
identified  knowledge  management  practice  as 
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the  preferred  solution  option  to  identify  and 
reuse  methodologies  and  measures. 

An  Architecture  of  Systems,  Software  and 
Models  to  Support  Analysis 

The  group  developed  an  “architecture”  of 
systems,  software  and  models  that  was  perceived 
to  be  necessary  to  meet  the  needs  of  military 
leaders,  and  defense  managers  in  the  twenty-first 
century.  This  architecture  is  shown  in  Figure  5. 
The  kinds  of  unprecedented  connectivity  and 
consistency  that  are  envisioned  are  required  to 
keep  analysis  and  analysis  products  relevant 
inside  the  short  decision  cycles  that  are 
emerging.  It  is  also  necessary  to  achieve  a 
culture  of  “Knowledge  Management”  in  which 
the  system  of  systems  we  are  attempting  to 
analyze  has  achieved  a  complexity  that  is 
beyond  the  comprehension  of  any  one  person  or 


Figure  5.  The  Future  “Framework”  for  Analysis 

There  are  three  primary  nodes  in  this  network  of 
systems.  First,  there  are  near  real-time  C4ISR 
systems.  Operational  planners  and  commanders 
who  are  tasked  to  execute  the  nation’s  military 
missions  use  these  systems.  In  the  “information 
age”  military,  they  exist  in  some  form  or  another 
from  the  National  Military  Command  Center 
(NMCC),  down  to  the  soldier  in  the  foxhole. 

The  second  node  is  analysis  models  and 
simulations.  These  models  and  simulations, 
ranging  from  simple  spreadsheets  and  decision 
tables  to  large  theater  simulations  are  used  to 
develop  and  evaluate  plans,  choose  courses  of 


action,  and  manage  the  DoD.  The  final  node 
envisioned  consists  of  training  simulations. 
Again,  these  simulations  vary  in  scope  and 
complexity. 

All  of  these  nodes  have  a  common  requirement. 
They  all  need  data  (both  primitive  and  meta)  to 
function.  If  we  are  to  achieve  the  envisioned 
interoperability  it  is  absolutely  essential  that  this 
data  be  logically  consistent,  and  portable  across 
domains  when  desired. 

One  fact  that  emerged  is  that  the  C4I  community 
is  embedding  decision  aides  into  emerging 
command  and  control  systems.  These  decision 
aids  are  effectively  models  that  operational  staff 
members  and  commanders  will  use  to  make 
decisions  on  those  short  timelines  we  see 
emerging.  It  is  important  that  analysts  (and 
others)  be  involved  in  the  development  of  these 
algorithms  to  increase  the  likelihood  that  the 
underlying  logic  is  sound  and  consistent. 
Moreover,  this  kind  of  involvement  is  necessary 
if  the  DoD  is  to  achieve  its  objectives  with 
respect  to  the  reuse  of  software,  and  achieve  a 
more  cost  effective  maintenance  stream  for 
models  and  data  of  all  types. 

In  theory  the  DU  COE  standards  provide  vehicle 
for  data  communication  from  live  systems  to 
simulations.  Flowever,  it  was  learned  that  the 
communication  between  the  C4I  and  analytic 
communities  will  not  support  this  objective. 
Moreover,  there  are  inconsistencies  between  the 
C4I  and  training  domains. 

An  example  illuminates  how  this  system  would 
improve  our  operational  decision  making. 
Envision  a  requirement  for  PACOM  to  deploy 
humanitarian  assistance  to  East  Timor.  There  is 
a  need  to  provide  logistical  infrastructure  to 
rebuild  or  secure  this  nation.  Also,  assume  the 
planning  process  produced  a  contingency  plan 
that  includes  identification  of  the  forces  that 
would  be  sent  on  this  mission. 

The  plan  and  all  of  its  assumptions  have  been 
described  in  some  analysis  model  (perhaps  in  an 
Operations  Plan  (OPLAN)  Format)  and  reside  in 
data.  At  the  time  the  requirement  to  execute  is 
identified,  the  data  in  the  plan  are  compared  to 
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ground  truth  data.  The  army  database  on  current 
forces  identifies  the  fact  that  the  25th  Supply 
and  Transportation  Battalion  which  was  in  the 
plan  is  not  available  because  it  is  deployed. 

And,  the  US  Transportation  Command 
(TRANSCOM)  data  shows  that  the  C-5s  that  are 
necessary  to  meet  the  timeline  are  committed  to 
some  other  high  priority  mission.  What  has 
happened  is  that  we  have  used  existing  data  to 
quickly  illuminate  assumptions  that  were  valid 
at  planning  time,  but  are  not  at  execution  time. 
Operators  are  then  quickly  able  to  focus  on 
managing  the  exceptions. 

The  deliberations  in  the  group  revealed  that 
there  are  fundamentally  no  significant 
technological  impediments  to  achieving  this 
vision.  The  obstacles  are  largely  related  to 
achieving  a  sound  common  data  model,  and 
developing  a  cultural  will  to  expose  data  to  other 
domains. 

Description  and  Discussion  of  WG  4  Findings 
and  Recommendations 

The  working  group  had  five  broad 
recommendations  associated  with  the  analysis  of 
new  tools  and  methods  for  analysis. 

•  Increase  DoD  emphasis  on  supporting  inter¬ 
service  collaboration  toward  standards.  This 
builds  on  the  existing  service  efforts. 
Existing  organizational  models  include  the 
Joint  Technical  Coordination  Group’s 
(JTCG)  (JMEM)  and  the  Modeling  and 
Simulation  Resource  Repository  (MSRR) 
Board  of  Directors.  This  should  include 
Joint  rules  for  reconciliation  of  standards.  A 
baseline  should  be  established  (i.e.  a 
minimal  standard  for  meta-data). 

•  Use  the  accumulated  but  limited  experience 
as  a  foundation  for  analysis  knowledge 
management  implementation.  Establish  a 
formal  plan  for  achieving  the  vision 
including  senior  leader  support  and 
involvement,  a  deliberate  investment 
strategy,  and  one  that  uses  the  multi¬ 
disciplinary  team.  There  should  be  a  co¬ 
evolution  of  culture  and  funding  for 
technology. 


•  The  analysis  community  should  be 
represented  in  the  DII  COE  process.  The 
DII  COE  standards  should  be  impacted  by 
requirements  that  emanate  from  the  analysis 
community. 

•  DoD  should  develop  and  resource  a  strategy 
for  collecting  “other  source”  data,  and 
applying  knowledge  management.  Other 
source  data  include  National  Training 
Center  (NTC)  exercise  data,  historical 
deployment  data  and  lessons  learned  data. 

•  Encourage  the  creation  and  use  of  “quick 
and  dirty”  tools.  This  should  be 
accompanied  by:  the  development  of  rules 
of  the  road  for  using  data,  the  accumulation 
of  best  practices  and  mechanisms  for  timely 
peer  review,  the  use  documentation  and 
allowance  for  evolution. 

The  Synthesis  Group  Report 

Vision  Statement:  Since  analysis  is  both  a 
product  and  a  process,  our  vision  is  a  re¬ 
emphasized  and  revitalized  practice  of  analysis 
to  more  effectively  and  more  efficiently  connect 
the  decision  maker  and  analyst  so  that  the  final 
product  is  not  the  primary  result. 

The  Synthesis  Group  included  members  from 
each  working  group  and  met  daily  with  the 
chairs  and  co-chairs  of  those  groups.  This  was 
done  to  both  highlight  common  areas  and  to 
request  effort  on  specific  topics  that  would  help 
the  Synthesis  team  in  their  efforts  to  create  a 
follow-on  action  plan  to  help  the  military 
operations  research  community  achieve  the 
overall  vision  of  the  conference. 

Toward  this  end,  the  Synthesis  Group  asked 
each  WG  to  articulate  a  vision  that  would 
capture  the  end  state  resulting  from  full 
implementation  of  their  group’s 
recommendations.  Based  on  their  inputs,  the 
Synthesis  Group  articulated  its  own  Vision 
Statement.  The  Synthesis  Group  believes  this 
vision  transcends  and  incorporates  the  subject 
area  working  groups’  visions  and  describes  how 
emerging  capabilities  such  as  collaboration  and 
knowledge  management  can  transform  the 
practice  of  military  analysis.  The  emergence  of 
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powerful  communication  and  computer-based 
capabilities  is  already  changing  the  economic 
world.  Where  before  the  focus  was  on 
production,  the  emerging  giants  of  e-commerce 
are  now  succeeding  through  their  acquisition 
and  management  of  information.  The  most 
successful  and  most  profitable  of  these 
businesses  are  those  that  enable  the  highest  level 
of  information  sharing,  so  that  opportunities  are 
quickly  identified,  decisions  made  based  on  the 
full  amount  of  information  and  insight  available 
in  the  work  force,  and  then  action  initiated 
appropriately.  The  knowledge  resident  in  every 
worker  is  rapidly  connected  to  the  decision 
makers  so  that  timelines  are  reduced  and 
customer  needs  satisfied,  optimizing  company 
effort  and  taking  full  advantage  of  the 
knowledge  base  of  the  organization.  In  a  like 
manner,  military  analysis  in  the  future  will  need 
to  be  fully  integrated,  with  analysts  and  decision 
makers  so  closely  linked  through  technical  and 
personnel  means  that  the  maximum  benefit  will 
flow  to  the  organization.  In  fact,  the  interaction 
will  be  such  that  the  final  results  of  any  study 
will  surprise  no  one,  since  decision  makers  and 
analysts  will  both  be  engaged  throughout  the 
analytic  process,  allowing  the  strengths  and 
insights  of  each  to  be  fully  incorporated.  Such  a 
construct  will  revitalize  the  practice  of  analysis, 
take  advantage  of  the  emerging  capabilities,  and 
better  address  new  challenges  confronting 
analysis  in  the  new  millennium. 

The  Tool  and  Challenge  Matrix 

The  products  of  WG  3  and  4  analysis  were 
combined  in  Figure  6  to  illustrate  what  new  or 
emerging  tools  and  techniques  show  promise  to 
address,  at  least  in  part,  the  challenges  identified 
by  WG  3. 

Those  challenges  fell  into  six  broad  categories: 

•  Changing  characteristics  of  warfare  in  the 
21st  Century  (New  Wars). 

•  Inclusion  of  the  effects  of  humans  and 
human  factors  (e.g.,  moral,  training,  etc.) 
into  models  (Behavioral). 


•  Showing  the  effect  of  logistics  and  other 
support  activities  on  warfighting  outcomes 
(support). 

•  Identification  of  complex  interrelationships 
between  input  variables  and  between 
internal  variables  (processes)  within  models 
(interactions). 

•  Explicit  quantification  of  risk  and 
uncertainty  in  analysis  and  in 
communications  with  decision  makers 
(risk). 

•  The  challenge  of  developing  and  gaining 
acceptance  of  analysis  techniques  that  are 
not  dependent  upon  traditional  simulation  or 
other  models  (e.g.,  linear  programs)  (non 
model). 
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Figure  6.  Tool  /  Challenge  Matrix 


WG  4  identified  many  recommendations 
regarding  new  or  emerging  tools  and  techniques 
that  should  be  explored  and  implemented  where 
appropriate.  These  tools  and  techniques  fell  into 
ten  categories  as  shown  in  Figure  6. 

The  first  category  listed  involves  standards. 
There  are  some  relatively  new  efforts  underway 
to  develop  common  standards  that  can  be 
applied  to  the  models  and  data  used  to  support 
analysis;  the  Army  Standards  Category 
Coordination  process  is  one  example  of  this. 
Defense  Modeling  and  Simulation  Office 
(DMSO)  -sponsored  efforts  to  develop  common 
conceptual  models  and  the  MSRR  are  other 
examples.  The  group  recommended  that  the 
fundamental  algorithms  underpinning  many 
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major  simulations  in  use  be  examined  in  open 
(peer  review)  forums  and  reconciled.  Although 
a  commonly  accepted  set  of  algorithms  would  be 
desirable,  it  is  not  necessary  that  all  services  or 
other  users  agree  to  a  single  representation  of  a 
warfare  phenomenon.  We  can  agree  to  disagree, 
provided  the  differences  are  clearly  documented 
and  understood  and  made  available  to  the 
analysis  community. 

The  second  technique  identified  by  WG  4  is 
automated  data  generation.  The  biggest  single 
complaint  from  the  WG  was  the  lack  of  needed 
data  and  the  difficulty  and  cost  of  generating  or 
obtaining  data  even  when  it  is  available. 
Automated  means  of  generating  commonly  used 
data  would  increase  the  responsiveness  and 
reduce  the  cost  of  getting  the  data. 

The  “Analyst’s  Rucksack”  is  an  example  of 
using  quick  and  simple  tools  and  decision  aids  to 
fill  in  for  large  model  deficiencies  and  directly 
support  warfighters.  The  Workshop  endorsed 
the  development  of  tool  sets  of  this  nature. 

The  commercial  world  is  also  producing  an 
array  of  tools  that  have  great  utility  for 
operations  research  analysts.  Tools  which  have 
been  around  for  several  years,  but  are  becoming 
increasingly  sophisticated  and  easy  to  use, 
include  statistical  and  data  analysis  packages, 
linear  and  nonlinear  programming,  process 
simulation  and  decision  analysis  software. 
Emerging  products  include  neural  networks, 
Bayes  nets,  colored  petri  nets  and  other  software 
that  propagate  inferences  or  influences. 

There  is  recent  research  that  is  producing  some 
new  techniques  that  may  be  applied  to  fix,  in 
part,  some  of  the  areas  that  lack  tools  for 
analysis.  Agent-based  simulation  may  help  in 
the  analysis  of  doctrine,  procedures  and  risk 
sensitivity,  and  recent  mathematical  models 
have  been  developed  to  assist  in  the  simulation 
(and  real-world  execution)  of  fusion  between 
multiple  sensors  and  sensor  disciplines.  This  is 
not  an  exhaustive  list,  but  these  are  two  areas 
that  show  promise. 

Performing  analysis  in  the  21st  century  will 
likely  lead  to  the  resurgence  of  the  multi¬ 


disciplinary  team.  However,  this  team  will  have 
a  different  composition  that  the  teams  in  the 
early  days  of  military  operations  research,  which 
were  largely  formed  from  hard  sciences. 
Although  some  problems  may  still  require 
physicists,  chemists,  etc.,  most  work  will  require 
groups  pulled  from  three  broad  areas.  The  first 
consists  of  domain  experts,  which  will  often 
include  warfighters,  doctrinal  experts  and  other 
expertise  in  military  operations.  The  second 
group  is  software  and  computing  experts,  as 
models  and  tools  become  increasingly  complex 
from  a  software  perspective.  The  third  group, 
which  helps  to  meld  the  other  two  together, 
consists  of  operations  research  analysts  and 
other  experts  from  related  analysis  domains, 
such  as  mathematics  and  statistics.  We  should 
no  longer  expect  that  a  single  military  operations 
research  analyst  be  able  to  read  and  write  code, 
develop  mathematical  models  and  possess  a 
broad  military  background  to  include  joint 
operations. 

The  implementation  of  knowledge  management 
techniques  identified  by  WG  1  and  presented  by 
plenary  session  and  lunchtime  speakers  will  also 
help  address  the  challenges  to  the  practice  of 
military  operations  research.  These  concepts  are 
developed  in  detail  in  other  sections  of  this 
report. 

Modeling  and  Simulation  Executive  Agents 
(MSEA)  have,  in  some  areas,  greatly  advanced 
the  state  of  the  art  in  modeling  and  simulation 
(M&S).  The  environmental  MSEAs  have  had  a 
great  degree  of  success  and  can  serve  as 
exemplars  of  the  process  that  leads  to  significant 
improvements.  They  have  developed  data 
standards  applicable  to  a  wide  variety  of 
simulations  (to  include  both  training  and 
analysis  simulations)  and  real  world  command 
and  control  systems,  and  populated  these  data 
structures  with  data  that  is  easily  obtained 
through  user-friendly  business  processes.  They 
have  also  developed  algorithms  to  transform  the 
data  into  information  useful  to  aggregated 
models;  for  example,  the  Army  has  developed 
techniques  for  automatically  generating  road 
networks  from  detailed  NATO-standard 
mobility  networks. 
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One  area  that  can  be  addressed  through  good 
knowledge  management  practices  involves 
“other  source”  information.  This  data  includes 
NTC  exercise  data,  historical  deployment  data 
and  “lessons  learned”  data.  These  data 
sometimes  exist  but  are  not  necessarily  recorded 
for  posterity  (e.g.,  much  NTC  information  is 
outbriefed  to  the  unit  commander  and  then 
deliberately  destroyed),  not  captured  at  all  (e.g., 
historical  deployment  data),  or  are  captured  only 
qualitatively  rather  than  quantitatively  (e.g., 
lessons  learned).  These  data,  if  resources  are 
applied  for  capture  and  storage,  can  significantly 
improve  our  ability  to  validate  models  against 
“real  world”  experiences.3 

Finally,  the  group  recommended  that  additional 
research  be  made  in  the  area  of  automated 
decision  making.  Large  simulation  models 
require  some  automated  representation  of  a 
human  decision  maker  (e.g.,  a  commander  and 
staff  in  an  analysis  model  or  an  opposition  force 
(OPFOR)  or  subordinate  commander  in  a 
training  model)  to  provide  reproducibility  or 
reducing  exercise/training  costs.  However,  the 
current  implementations  are  recognized  to  be 
insufficient  and  unrealistic.  Automated  decision 
making  can  also  be  embedded  in  real-time 
decision  aids,  not  to  replace  a  human  decision 
maker  but  to  advise  him  or  her. 

Analysis  to  Support  Decision  Making 

Figure  7  illustrates  one  view  of  the  current 
practice  of  analysis. 


3  It  is  recognized  that  exercise  data  is  not  real  combat  data, 
but  it  is  useful  surrogate  data  that  may  be  more  accurate 
than  outputs  from  abstract  models  and  simulations. 


Figure  7.  Analysis  to  Support  Decision  Making 


It  shows  that  analysis  is  focused  on  one  goal: 
helping  decision  makers  choose  among 
alternatives.  The  four  activities  analysts  support 
in  this  practice  include: 

•  Helping  the  decision  maker  define  the  issue 
or  questions. 

•  Helping  to  define  the  decision  space  so  that 
a  full  range  of  alternatives  can  be 
considered. 

•  Establishing  meaningful  measures  of 
effectiveness  and  criteria  that  will  help  show 
the  strengths  and  weaknesses  of  the  various 
alternatives. 

•  Then  creating  understandable,  traceable  and 
repeatable  methodologies  that  will  assist  in 
this  selection  process. 

To  do  this,  an  analytic  foundation  must  be  in 
place.  This  foundation  includes: 

•  People  who  contribute  to  the  process, 
primarily  analysts,  but  including  a  broad 
range  of  other  scientists,  engineers  and 
SMEs. 

•  Models,  or  the  concepts,  mathematical 
abstractions  or  other  constructs  that  are  used 
to  decompose  a  problem  and  focus  on  key 
aspects. 

•  Data,  scenarios,  assumptions  and  other 
factors  that  are  necessary  to  support  the 
process. 
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The  issue  addressed  by  this  workshop  is  how 
this  framework  is,  and  will  be,  impacted  by  the 
changes  in  the  national  security  and 
technological  landscapes  and  how  the  practice  of 
analysis  should  evolve  to  handle  the  challenges 
and  take  advantage  of  the  opportunities  provided 
by  these  changes. 

Of  course,  the  very  complexities  presented  by 
the  rapidly  changing  landscapes  associated  with 
this  environment  may  have  removed  the  ability 
for  analysts  and  others  to  focus  on  the  big 
picture.  The  new  environment  as  assessed  by 
WG  3  presents  a  conundrum:  we  are  asked  to 
provide  insight  on  the  complex  interrelationships 
between  factors  input  to  and  within  our  models 
(that  reflect  interrelationships  in  the  real  world), 
but  we  are  also  asked  to  reduce  the  complexity 
and  increase  the  transparency  of  our  tools  and 
analysis.  Although  it  may  not  be  possible  or 
necessary  to  reduce  the  complexity  of  our  high- 
end  models  (recognizing  of  course  that  not  all 
analysis  requires  large  and  complex  models), 
open  sharing  of  model  data,  assumptions, 
algorithms,  etc.,  all  within  a  set  of  agreed-upon 
standards  and  implemented  through  knowledge 
management,  may  increase  the  transparency  of 
the  models  and  analysis  process  among  trained 
analysts.  The  analyst  will  always  have  to 
convert  complex  mathematical  and  structured 
logic  into  forms  understandable  to  a  decision 
maker  without  an  operations  research 
background  in  all  but  the  simplest  cases.  Thus, 
an  elegant  solution  may  not  be  a  reduction  in 
complexity  to  a  handful  of  variables,  but  instead 
be  a  transparency  and  sharing  among  analysts 
that  is  communicated  to  the  decision  maker 
throughout  the  analysis  (rather  than  at  the  end) 
in  a  clean  and  direct  (if  not  elegant)  manner. 

At  the  same  time,  there  are  potential  solutions 
available,  at  least  to  address  part  of  the  changing 
environment,  and  to  help  our  analytic 
community  and  other  parts  of  the  DoD 
enterprise  to  adapt  and  prevail  in  the  emerging 
environment  of  the  21st  century.  Dr  Ruth 
David,  the  first  keynote  speaker  for  this 
workshop,  defined  Knowledge  Management  as: 

•  The  process  through  which  an  enterprise 
uses  its  collective  intelligence  to  achieve 


strategic  objectives.  Furthermore, 
Knowledge  Management  is  realized  through 
strategy,  people,  process  and  technology. 

Inherent  in  her  definition  was  the  understanding 
that  the  enterprise  using  knowledge  management 
was  a  “lean,  mean,  fighting  machine”  team 
focused  on  common  objectives  and  with  a 
shared  understanding  of  the  necessity  and  value 
of  each  other’s  contribution.  Such  a  definition 
of  enterprise  may  not  reflect  the  current  state  of 
DoD,  where  mutual  contributions  are  not 
recognized  and  parochialism  can  break  down 
teamwork.  However,  to  the  extent  that  shared 
goals  and  teamwork  reflect  DoD,  and  in  the 
areas  that  knowledge  management  can  help  the 
department  move  toward  this  enterprise  vision, 
the  department  should  surely  embrace 
knowledge  management  and  its  implementation. 

Certain  aspects  of  knowledge  management  can 
already  be  endorsed  in  ways  that  do  not  threaten 
existing  culture.  In  particular,  knowledge 
management  has  the  ability  to  help  us 
distinguish  between  what  we  know  and  what  we 
don’t  know  by  helping  the  enterprise  quickly  tap 
the  collective  knowledge  of  the  individuals  in 
the  organization.  Done  correctly,  such  a 
capability  will  allow  an  individual  assigned  to 
work  a  problem  the  capacity  to  poll  all  experts  in 
or  out  of  the  organization  to  assess  what  might 
already  be  known  or  what  is  in  the  process  of 
being  studied  about  the  problem.  Rather  than 
wasting  time  reinventing  wheels,  a  properly 
implemented  knowledge  management  system 
will  save  enormous  time  and  resources  by 
helping  the  analyst  and  others  rapidly  determine 
what  is  already  known  as  the  basic  starting  point 
of  any  study.  The  same  system,  by  enabling 
rapid  connectivity  between  analytic  practitioners 
will  also  enhance  information  sharing  and 
collaboration.  When  one  study  team,  already 
focused  on  a  specific  area,  realizes  that  the  value 
of  its  research  can  be  increased  by  adding  just  a 
small  amount  of  effort  in  order  to  address  a 
broader  range  of  questions  and  meet  the  needs  of 
a  broader  customer  base,  one  promise  of 
knowledge  management  will  be  realized.  In 
fact,  as  the  limited  resources  available  in  an 
organization  self-organize  to  focus  on  the  areas 
they  do  best  in  order  to  provide  a  broader  base 
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of  insight  and  information  for  the  organization, 
the  people  involved  will  clearly  understand  that 
their  value-added  is  not  in  developing  and 
hoarding  information,  but  rather  in  sharing 
information  in  ways  that  enhance  the  overall 
capabilities  of  the  enterprise.  Value-added 
comes  not  from  having  information  since  in 
today’s  world  information  will  be  available  to 
everyone.  The  value  instead  comes  from  the 
ability  to  use  available  information  in  new  and 
creative  ways  to  enhance  the  capabilities  of  the 
organization. 

Returning  to  our  analytic  construct,  we  can  see 
that  changes  in  the  national  security  landscape 
are  already  shaking  some  of  the  analytic  pillars. 
Evolving  from  attrition-based  warfare  to  effects- 
based  warfare,  understanding  the  implications  of 
Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance  and 
Reconnaissance  (C4ISR)  systems  and 
capabilities,  and  the  emergence  of  a  broad  new 
array  of  asymmetric  threats  are  all  impacting  the 
ability  of  the  analytic  community  to  develop 
reasonable  measures  of  effectiveness  as  well  as 
to  develop  clear  and  repeatable  methodologies  to 
assess  these  aspects  of  military  activity. 


Figure  9  provides  more  detail  on  the  changes, 
with  the  circled  “national  security  landscape” 
reflecting  the  circled  area  on  Figure  8.  We’ve 
already  hinted  at  the  changing  paradigm  of 
massed  armies  meeting  on  a  battlefield  to  the 
massing  of  effects  at  the  appropriate  time  and 
place  as  the  way  current  US  military  leadership 
is  starting  to  think  about  warfare.  In  fact,  to  help 
think  through  the  implications  of  such  changes, 
MORS  has  conducted  a  number  of  workshops 
and  will  hold  a  C4ISR  Follow-on  Workshop  in 
the  fall  to  continue  to  evolve  our  understanding 
in  this  arena. 


❖.National  Security  Landscape^,** 
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-  Massing  forces  ->  massing  effects  (C4ISR  follow-on 


meeting) 

-  Environmental  -  Threat  (World  is  changing  -  are  we?) 

-  Ash  Carter  &  Bill  Perry  (A/B/C  Lists  from  the  QDR 
Workshop) 

-  Technology  Insertion 

-  QDR  2001  (strategy  -  ???) 


•  Technology  Landscape 

-  Knowledge  Management 

-  Collaborative  Analysis  Management 

-  Commercial  Initiatives 


While  these  areas  are  outlined  with  the  dashed 
lines  (which  appear  in  red  in  color  versions  of 
Figure  8),  the  technological  landscape  and 
emergence  of  new  capabilities  like  knowledge 
management  are  impacting  the  rest  of  the  words 
to  potentially  transform  the  way  we  conduct 
analysis. 


Figure  9.  Challenges 

Some  of  our  speakers  in  this  workshop  have 
referred  to  the  Ash  Carter/Bill  Perry  work  on 
who  the  threats  of  the  future  are.  They  range 
from  an  A  List  of  direct  threats  to  national 
survival  which  would  include  homeland  attack 
by  catastrophic  terror  agents,  to  a  B  List  of 
threats  to  national  interests  that  the  current 
strategy  seems  to  address,  to  a  C  List  of 
important  problems  that  do  not  threaten  US  vital 
interests  (e.g.  Kosovo,  Somalia).  In  addition, 
new  technologies  require  new  analytic 
capabilities  to  measure  the  effects  and  potential 
contributions  from  such  areas  as  directed  energy, 
information  technologies,  unmanned  systems 
and  space.  The  QDR  process  may  change  the 
national  military  strategy,  so  the  analytic 
profession  will  need  to  be  responsive  both  in 
helping  to  determine  the  implications  of  strategy 
shifts  as  well  as  in  implementing  any  new 
strategy  in  the  analyses  of  the  future. 
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Just  as  the  national  security  landscape  is 
changing,  so  is  the  technology  landscape  in 
which  analysis  is  conducted.  This  workshop  has 
highlighted  knowledge  management  and 
collaborative  capabilities,  but  there  are  a  number 
of  other  initiatives,  such  as  simulation-based 
acquisition  and  other  e-commerce  activities  that 
the  analytic  community  needs  to  track  and  adopt 
where  it  makes  sense  to  do  so. 


Figure  10.  Potential  Future  “Framework” 

Figure  10  shows  an  illustration  of  an  ideal  end- 
state  for  the  current  efforts  to  improve  analytical 
practices.  There  are  three  families  of  models  and 
simulations:  M&S  embedded  in  real-time 
command  and  control  systems;  training  and 
exercise  support  simulations;  and  models  and 
simulations  used  to  support  analysis  (to  include 
Test  and  Evaluation  (T&E)  analysis  and  analysis 
of  non-warfighting  issues  such  as  infrastructure 
and  business  process  improvement).  Within  each 
domain  are  a  set  of  models  and  simulations 
tailored  to  the  purpose  at  hand.  No  single  model 
will  apply  universally  across  all  domains,  but  in 
this  ideal  end  state  we  have  data,  software 
objects  and  process  representations  shared 
across  the  domains  where  it  is  logically 
consistent  with  the  model  and  purpose. 

Infrastructure  programs  currently  under 
development  are  assumed  to  be  in  place  in  this 
end  state,  to  include  the  High-Level  Architecture 
(HLA)  for  simulations  and  the  DU  COE  linking 
all  areas  together.  HLA  federations,  which  may 
include  simulations  widely,  distributed  over 
space,  will  exist  where  appropriate,  but  these 
federations  should  be  standardized  and  validated 


for  consistency  across  federates  before  use.  The 
creation  of  “on  the  fly”  federations  that  grab 
objects  and  federates  scattered  across  the  DII  to 
create  unique  environments  is  probably  still  not 
possible  in  this  end  state  because  there  is  no 
guarantee  that  the  representations  will  be 
internally  consistent  (consistency  is  not  assured 
even  if  data  is  passed  through  HLA),  and  it  is 
unlikely  that  a  single  analyst  or  team  located  at  a 
single  site  will  be  familiar  with  all  of  the  objects 
or  federates  potentially  available.  Instead,  these 
simulations  may  be  combined  through  the  use  of 
collaborative  analysis  rather  than  software 
linkages. 

All  of  the  tools  are  presumed  to  be  in  a  bed  of 
warm  data.  That  is,  there  is  a  data  infrastructure 
that  makes  data  readily  available  (with 
exhaustive  metadata  explaining  the  derivation 
and  accredited  use  of  the  data)  for  all  of  these 
systems.  For  example,  an  analysis  of  real-world 
capability  should  be  able  to  draw  on  the  same 
data  set  showing  current  status  and  deployments 
of  units  that  a  real-time  command  and  control 
system  might  use.  Similarly,  an  analysis  of  a 
future  capability  (for  example,  a  2010  force  in  a 
particular  scenario)  should  be  able  to  draw  on  a 
validated  and  commonly  used  set  of  data  to 
represent  the  scenario,  system  performance, 
OPFOR  performance,  etc.  Generation  and 
storage  of  this  data  requires  a  significant 
resource  commitment  within  DoD,  but  it  is 
necessary  to  achieve  the  desired  end  state. 

A  key  feature  of  this  environment  is  that  the 
output  of  the  models,  as  well  as  the  results  of 
specific  analyses  and  the  lessons  learned  from 
training  and  exercises  are  also  added  to  the 
common  database.  This  provides  a  useful 
source  of  information  that  can  be  of  particular 
help  to  quick  turn-around  analyses,  which  may 
synthesize  the  outputs  of  previous  models, 
exercises  and  analyses. 

The  Action  Plan 

The  Synthesis  Group  identified  a  number  of 
actions  that  should  become  part  of  a  long-range 
action  plan  that  the  community  should  consider 
and  flesh  out  over  the  next  few  years  as  it 
evolves  the  practice  of  military  analysis  in  DoD. 
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There  are  several  steps  that  MORS  itself  can 
implement  immediately  that  have  little  work 
impact  but  enormous  potential  to  improve  the 
ability  of  this  community  to  connect  with  each 
other  and  better  meet  the  over  all  needs  of  its 
membership.  In  addition  to  the  items  listed 
below,  the  working  groups  provided  a  number  of 
general  and  specific  steps  that  will  help  the 
community  learn  to  harness  knowledge 
management  and  collaboration  as  well  as  to 
address  emerging  issues  in  the  changing  security 
environment.  Generalized  recommendations 
endorsed  by  the  workshop  include  working  the 
people,  process  and  technology  problems  in 
order  to  ensure  a  better  final  product. 

Actions  for  MORS  include: 

•  Near  Term: 

Update  the  MORS  membership  directory 
and  database  with  more  information  on  each 
member. 

Evolve  the  role  of  the  Electronic  Media 
Committee  to  address  enterprise  knowledge 
management. 

Contribute  to  a  Code  of  Best  Practices  for 
Analysis  and  Joint  Experimentation. 

•  Mid  Term: 

Undertake  focused  workshop  activity  on  this 
topic. 

•  Far  Term: 

Use  the  Education  Colloquia  to  address 
ideas  that  Knowledge  Management  is  an 
important  element  of  Professional 
Development 

Actions  for  DoD  include: 

The  working  group  recommendations 
provide  a  phased  approach  to  knowledge 
management  and  collaboration 
implementation  by  focusing  on  people, 
process,  technology  and  product,  as  well  as 
list  challenges  and  promising  tools  and 
methodologies  to  meet  those  challenges. 

Table  4  summarizes  the  focus  and  potential 
impact  of  the  specific  actions  recommended. 


_ Summary  Recommendations _ 

Vision/Policy:  Develop  and  promulgate  an  Enterprise 

Vision  and  strategic  directions. _ 

Culture:  Take  action  to  remove  deeply  rooted  cultural 
barriers  (e.g.,  remove  the  incentives  to  hoard 

knowledge) _ 

Organization:  Ensure  that  there  is  buy-in  by  Top 
Management.  Investigate  organizational  constructs 
including  business  rules  that  best  enable  global 

collaboration  of  military  analysts. _ 

People:  Provide  incentives  to  analysts  to  collaborate 

and  share  knowledge. _ 

Process:  Institutionalize  processes  to  enhance 
collaboration  and  the  sharing  of  tacit  and  explicit 

knowledge  among  the  analytical  community. _ 

Analytic  Tools: _ 

•  Develop  and  emphasize  joint  standards  in 

algorithm  and  data  development. _ 

•  Expand  the  development  and  maintenance  of  data 

warehouses. _ 

•  Focus  basic  research  and  tool  development  to 

meet  emerging  challenges _ 

Table  4.  Summary  Recommendations 

First,  cultural  changes  are  a  necessary 
prerequisite  to  effective  transformation  of  the 
analysis  community  (e.g.,  the  need  to  dispel  the 
myth  that  “knowledge  is  power”).  Second  are 
several  organizational  issues  that  must  be 
addressed.  One  of  these  is  to  ensure  top 
management  buys  in  to  this  transformation. 
Another  includes  reassessing  how  security  issues 
can  be  addressed  in  an  information-centric,  open 
environment.  In  the  third  area,  decision  makers 
must  recognize  that  the  analysts  involved  in  all 
phases  of  the  assessment  process  constitute  a 
critical  intellectual  resource. 

They  must  be  given  incentives  to  stimulate  them 
to  contribute  and  draw  from  future  knowledge 
warehouses.  Tying  all  this  together  are  the  links 
between  the  vision  and  the  high  level  guidance 
to  the  analysis  process  and  the  primary  products 
of  that  process  (i.e.,  effective,  timely,  rigorous 
analyses).  To  implement  this  linkage,  a  set  of 
enhanced  processes  is  needed  (e.g., 
institutionalized  processes  to  diffuse  tacit  and 
explicit  knowledge)  supported  by  a  broad  set  of 
tools  (e.g.,  knowledge  warehouses,  collaboration 
tools).  These  processes  and  tools  are  constrained 
by  available  resources  (e.g.,  funds,  manpower) 
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such  that  a  special  funding  mechanism  may  be 
required  to  satisfy  time-urgent  joint  needs.  In 
particular,  basic  research  must  be  continued  in 
those  areas  where  understanding  of 
phenomenology  is  not  available  (human  decision 
processes,  Information  Operations/Information 
Warfare  (IO/IW),  etc)  as  well  as  the  continuing 
development  of  analysis  tools  to  address  the 
emerging  challenges. 

The  Synthesis  Group  feels  that  by  addressing  the 
needs  of  the  community  to  implement  our 
vision,  the  practice  of  analysis  can  evolve  from 
its  current  position  of  a  limited  toolkit  and 
methodologies  to  address  a  limited  number  of 
issues  to  a  complete  tool  set  connecting  analysts 
and  decision  makers  to  address  the  full  array  of 
national  security  challenges. 

This  workshop  has  demonstrated  that  there  is 
great  potential  to  be  realized  from  the  harnessing 
of  the  knowledge  in  our  heads  and  in  the 
systems  that  support  our  analytical  efforts.  In 
addition,  there  is  great  power  in  undertaking  a 
more  collaborative  approach  to  the  analyses  that 
we  perform.  The  combination  of  enterprise 
knowledge  management  and  collaboration  can 
deliver  better  analysis  products  faster. 


Summary 

The  way  ahead  to  realize  this  vision  will  be 
difficult.  Nothing  less  than  a  cultural  change  is 
required.  The  demands  of  the  national  security 
environment  require  that  we  find  ways  to  alter 
the  information  security  environment  to  permit 
knowledge  sharing  and  collaboration  on  an 
enterprise  scale.  Getting  analysts  to  collaborate 
and  breaking  down  the  barriers  to  that 
collaboration  and  knowledge  sharing  are 
fundamental  changes  that  require  bottom  up  as 
well  as  top  down  change. 

The  members  of  this  workshop  acknowledge  the 
potential  value  of  such  change.  It  is  now  up  to 
us  to  obtain  the  buy-in  of  the  DoD  leadership  by 
convincing  them  that  the  benefits  are  worth  the 
investment  of  time,  money  and  energy  to  realize 
the  vision  articulated  by  this  workshop. 
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Appendix  A:  Acronyms 


ACATI 

Acquisition  Category  I 

AFSAA 

Air  Force  Studies  and  Analysis 

AoA 

Analysis  of  Alternatives 

ARL 

Army  Research  Laboratory 

ASD(C3I) 

Assistant  Secretary  of  Defense  for  Command,  Control, 

ASD(S&TR) 

Communications  and  Intelligence 

Assistant  Secretary  of  Defense  for  Strategy  and  Threat  Reduction 

C4I 

Command,  Control,  Communications,  Computers  and  Intelligence 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelligence, 

CAA 

Surveillance  and  Reconnaissance 

Center  for  Army  Analysis 

CEO 

Chief  Executive  Officer 

CINC 

Commander  in  Chief 

CNO 

Chief  of  Naval  Operations 

CONOPS 

Concepts  of  Operations 

COTS 

Commercial-Off-The-shelf 

CP  AM 

DII  COE 

Defense  Information  Infrastructure  Common  Operating  Environment 

DMDC 

DMSO 

Defense  Modeling  and  Simulation  Office 

DUSD  (P&R) 

EUCOM 

US  European  Command 

EXCMS 

Executive  Committee  for  Modeling  and  Simulation 

FBE 

Fleet  Battle  Experimentation  Program 

FFRDC 

Federally  Funded  Research  and  Development  Center 

GIG 

Global  Information  Grid 

HLA 

High  Level  Architecture 

10 

Information  Operations 

ISR 

Intelligence,  Surveillance  and  Reconnaissance 

IW 

Information  Warfare 

IWAR/CPAM 

IWARS 

IWEDA 

Integrated  Weather  Effects  Decision  Aid 

J-8 

Director,  Force  Structure  Resource  and  Assessment 

JDS 

Joint  Data  Support 

JFCOM 

US  Joint  Forces  Command 

JMEM 

Joint  Munitions  Effectiveness  Manual 

JSF 

Joint  Strike  Fighter 

JTCG 

Joint  Technical  Coordination  Group 

JTF 

Joint  Task  Force 

KM 

Knowledge  Management 

M&S 

Modeling  and  Simulation 

MEDEVAC 

Medical  Evacuation 

MOOTW 

Military  Operations  Other  Than  War 

MORS 

Military  Operations  Research  Society 

MRS-05 

Mobility  Requirements  Study  -  2005 

MSEA 

Modeling  and  Simulation  Executive  Agents 

MSRR 

Modeling  and  Simulation  Resource  Repository 
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NATO 

North  Atlantic  Treaty  Organization 

NMCC 

National  Military  Command  Center 

NTC 

OA 

National  Training  Center 

OASD(C3I) 

Office  of  the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications  and  Intelligence 

ODPA&E 

Office  of  the  Director  for  Program  Analysis  and  Evaluation 

OMFTS 

Operational  Maneuver  From  The  Sea 

OPFOR 

Opposition  Force 

OPLAN 

Operations  Plan 

OPSEC 

Operations  Security 

OPTEMPO 

Operations  Tempo 

OSD  (PA&E) 

Office  of  the  Secretary  of  Defense  (Program  Analysis  and 
Evaluation) 

PACOM 

US  Pacific  Command 

PERSTEMPO 

Personnel  Tempo 

POM 

Program  Objective  Memorandum 

PPBS 

Planning,  Programming  and  Budgeting  System 

PTD 

Phased  Threat  Distribution 

QDR 

Quadrennial  Defense  Review 

SECNAV 

Secretary  of  the  Navy 

SIPRNET 

SECRET  Internet  Protocol  Router  Network 

SME 

Subject  Matter  Expert 

T&E 

Test  and  Evaluation 

TPFDD 

Time-Phased  Force  Deployment  Data 

TRANSCOM 

US  Transportation  Command 

WG 

Working  Group 
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Working  Group  1  Presentations 


♦  Mr.  Michael  Yoemans,  Introduction  to  Knowledge  Management 

♦  Mr.  Gary  Coe,  The  Analytic  Enterprise 

♦  Mr.  Robert  Graebener,  KM  Lessons  from  STOW 

♦  Mr.  John  Black,  A  Vision  of  an  Analytic  Portal 

♦  Mr.  Jack  Sheehan,  DMSO  Initiatives  to  Identify,  Access  and  Store 
Knowledge 


^  MILITARY  OPEEMI^S  M5EARCH  SOCIETY^ 


WG  #1  Management  of  Analytic  Knowledge 
Focus  Questions 

♦  How  is  analytic  knowledge  currently  discovered,  captured, 
expanded,  organized,  and  shared  within  the  DoD  analysis 
community? 

♦  What  Knowledge  Management  techniques  can  be  applied  to 
improve  DoD  analysis? 

♦  What  analytic  processes  would  be  impacted  by  the 
implementation  of  Knowledge  Management? 

♦  What  are  the  obstacles  in  successfully  implementing 
Knowledge  Management  to  the  DoD  Practice  of  Analysis? 

♦  What  recommendations  do  you  have  to  improve  the  existing 
analytic  practice  using  Knowledge  Management? 

(S3  _ 
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WG  #1  Management  of  Analytic  Knowledge 

The  Workgroup’s  Vision 
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WG  #1  Management  of  Analytic  Knowledge 

The  Strategy 


♦  DoD  Leadership  by  J-8  and  OASD(C3I)  to  establish  policies 
and  facilitate  change. 

♦  Change  the  mindset  of  senior  leaders  and  culture  in  DoD  to 
promote  collaboration  and  knowledge  sharing. 

♦  Partnering  of  appropriate  DoD  organizations  and  agencies  to 
overcome  the  barriers  to  knowledge  management 
implementation. 

♦  Continuing  education  and  training  in  knowledge  management 
practices  and  technologies. 

♦  Prototyping  of  knowledge  management  implementation  to 
important  problems  to  evolve  solutions. 
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WG  #1  Management  of  Analytic  Knowledge 

Current  Use  of  Analytic  Knowledge  in  DoD  Analysis  Community 
“SILOED”  KNOWLEDGE 

♦  Information  “silos”  determined  by  organizational  boundaries 
— division,  functions,  etc. 

♦  Studies  centered. 

♦  Navigation  via  operating  and  file  systems. 

♦  Reliant  on  internally  generated  information. 

♦  Information  shared  via  email  attachment. 


GS1 


Current  analysis  is  done  in-house  using  information  available  inside  the 
organization.  Existing  information  sources  are  designed  to  meet  functionally 
oriented  information  needs  —  not  for  doing  operational  analysis. 
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WG  #1  Management  of  Analytic  Knowledge 

Knowledge  Management  Techniques  (Page  1  of  3) 


♦  Information  shared  via  email 
;  attachment.  ; 


ENTERPRISE  KNOWLEDGE 

♦  Enterprise-wide  integration  of 
information  resources. 

♦  Issues  centered  (Information, 
people  and  tools). 

♦  Navigation  via  hypertext 
browser  links  and  search 


Integrates  external  information 
from  strategic  partners, 
suppliers,  customers  and  third 
party  sources. 

Information  shared  via 
knowledge  portals. 


As  contrasted  by  the  status  quo  in  a  knowledge  organization  analysts  would  use 
or  have  access  to,  enterprise-wide  knowledge  repositories  and  should  be  issues 
oriented,  leveraging  the  evolving  technologies  that  permits  board  navigation  of 
the  knowledge-base. 
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WG  #1  Management  of  Analytic  Knowledge 

Knowledge  Management  Techniques  (Page  2  of  3) 

Private  Sector 

♦  THESAURAL  BROWSERS 

Browsers  generate  lists  of  related  terms  and  resources,  allowing 
users  to  take  advantage  of  otherwise  unexpected  associations 
between  seemingly  unrelated  information  sources  and  types. 

♦  PERSONALIZED  INTRANET  PORTALS 

The  creation  of  customizable  intranet  interface  allows  analysts 
selective  access  to  information  resources  required  to  perform  their 
jobs,  enables  employee  automation  of  simple,  routine  tasks  and 
creates  a  vehicle  for  a  central  information-management  body  to 
“push”  relevant  content  to  distributed  users  in  real  time. 

♦  KNOWLEDGE  SHARING  INCLUSIVE  PERFORMANCE 
REVIEWS 

Incorporation  of  a  knowledge-sharing  into  the  corporate  culture 
with  incentives. 

dH 


Industry  has  provided  the  models  that  DoD  can  leverage.  A  study  recently  done 
by  the  Working  Council  for  the  Chief  Information  Officers  demonstrates  this 
point.  WG  #1  included  a  copy  of  this  material  as  a  part  of  their  proceedings. 
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WG  #1  Management  of  Analytic  Knowledge 

Knowledge  Management  Techniques  (Page  3  of  3) 

Private  Sector 

♦  FEDERATED  KNOWLEDGE  MANAGEMENT 

Best-practice  teams  in  analysis  divisions  or  analysis  units  leverage 
their  subject  expertise  to  discover  and  codify  best  practices,  which 
are  then  passed  on  to  a  corporate  knowledge-management  group 
that  facilitates  the  propagation  of  best  practices  across  the 
enterprise. 

♦  INFORMATION-EXPIRATION  MECHANISMS 

Intranet  content  is  automatically  identified  by  date  of  creation 
and/or  last  modification;  dated  documents  are  flagged  and  routed 
to  their  authors,  who  review  them  for  relevancy  and  decide  to 
archive,  delete,  modify  or  reinsert  the  documents  back  into  the 
corporate  database. 

♦  CONTENT- VALIDATION  PROTOCOLS 
A  process  to  prevent  the  propagation  of  inconsistent  or  incorrect 
information  throughout  the  enterprise. 
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WG  #1  Management  of  Analytic  Knowledge 

Analytic  Practices  That  Use  Analytic  Knowledge 


The  DoD  Analytic  Process 

Skilled  Labeled  by  Products 


Analysts 


± 


Data  Sources 
and  Validation 


Instrumentation 


DISSEMINATION  OF 
REPORT  WITH 
CONCLUSIONS  and 
RECOMMENDATIONS 


Plan  for  Data 


Collection 


In  order  to  implement  Knowledge  Management  (KM)  to  the  DoD  Analytic 
Process,  we  need  to  have  a  common  understanding  of  this  complex  process. 
This  was  the  working  groups  (1&2)  consensus  and  all  agreed  that  such  an 
understanding  was  critical  to  successful  application  of  KM  practices. 
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WG  #1  Management  of  Analytic  Knowledge 

Obstacles  to  Implementing  Knowledge  Management 

♦  Policy  makes  it  impossible  to  use  existing  knowledge  tools. 

♦  There  is  no  Global  Information  Grid. 

♦  The  pace  of  technological  change  is  high. 

♦  No  economic  model  that  provides  the  incentive  for  sharing. 

♦  Organizational  and  individual  resistance. 

-  Resistance  to  change. 

-  What’s  in  it  for  me? 

-  Is  this  “too  hard”? 

-  The  institutional  incentive  to  hoard  information  and  knowledge. 

♦  The  Challenges  of  Managing  Unstructured  Information  (e.g. 
analyses). 

♦  The  need  for  security  and  protection  of  information  in  the 
conduct  of  many  analyses. 

-  The  value  of  intellectual  property. 


The  working  group  envisioned  a  number  of  major  obstacles  that  would  need  to 
be  overcome  in  order  to  implement  KM  successfully  throughout  the  analytic 
community.  This  slide  shows  the  important  problem  areas  that  the  working 
group  tried  to  address  in  their  recommendations.  These  obstacles  were  carefully 
considered  in  the  formulation  of  the  strategy  that  the  working  group 
recommended  in  implementing  KM  to  the  analytical  community. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  1  of  7) 

♦  The  DoD  Sponsor  implement  knowledge  management  techniques, 
particularly  to  establish  an  intranet  portal,  in  support  of  an 
important  study  with  suitable  scope,  schedule,  and  content  that 
partners  DoD  components  Military  Operations  Research  Society, 
industry,  and  academia.  The  elements  that  should  be  considered 
along  with  strategy  already  discussed  include: 

-  People 

-  Process 

-  Technology 


This  is  our  overarching  recommendation,  which  detailed  in  the  next  series  of 
slides  under  the  headings  of: 

Strategy 

People 

Process 

Technology 

Specific  organizations  that  should  be  included  are  the  Services,  OASD(C3I), 
DTIC,  MSIAC,  DMDC  and  DMSO. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  Page  2  of  7 


♦  People 

-  DoD  and  MORS  lead  an  assessment  of  the  capability  to  provide  a 
baseline  inventory  of  analysts  for  DoD  operations  research  studies 

-  MORS  assess  its  capability  to  establish  a  Knowledge  Management 
Professional  Development  Program  for  the  DoD  Analytic 
Enterprise  that  would  include  the  identification  of  skills  and 
products,  the  refinement  of  skill  levels,  and  the  training  of 
analysts. 

-  The  DoD  sponsor  evaluate  successful  skill-inventory  systems  (e.g. 
Northrop-Grumman  and  Y2K)  for  implementation  within  the  DoD 
Analytic  Enterprise. 


Industry  experience  and  DoD  case  studies  show  that  people  is  one  of  the  most 
critical  success  factors  associated  with  KM.  The  recommendations  cited  here 
are  designed  to  provide  the  fundamental  foundation  for  organizing  and 
educating  people  using  KM  techniques  being  applied  by  industry.  Key  DoD 
organizations  that  could  be  involved  in  pursing  this  recommendation  include 
ASD  (C3I),  DUSD(P&R),  OSD(PA&E),  and  DMDC. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  3  of  7) 


♦  Process: 

J-8  accept  leadership  with  OASD(C3I)  of  an  evolving  analysis 
process  for  the  purpose  of  improving  analysis  in  a  future  analysis 
state.  Notionally,  the  evolving  process  could  be  closely  interfaced 
with  an  evolving  Knowledge  Management  Process  as  shown  on 
the  next  slide. 


Instrumentation 
Plan  for  Data 
Collection 

(US 


Skilled 

Analysts 
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The  working  group  recognized  that  the  future  (to-be)  process  incorporating  KM 
is  still  to  be  defined.  Nevertheless,  this  slide  helps  illustrate  the  important 
interrelationships  related  to  collaboration,  creation,  and  reuse  of  a  well 
constructed  analytical  KM  environment  and  extensible  knowledge-base. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  5  of  7) 


♦  Process  continued: 

-  The  DoD  determine  the  business  rules  for  contributing  analytic 
knowledge.  For  example,  what  are  the  current  rules  that  control 
the  access  to  past  studies  and  how  could  these  rules  be  changed  to 
improve  access  such  as  those  stored  at  Federally  Funded  Research 
and  Development  Centers  (FFRDC).  A  special  case  is  the 
identification  of  the  business  rules  for  administering  and 
controlling  an  analyst’s  portal. 

-  The  DoD  Sponsor,  with  the  assistance  of  MORS,  identify  new 
types  of  information  required  for  future  analyses  (e.g.  foreign 
demographics,  information  warfare  related,  economic  information, 
national  infrastructure  related,  chemical/biological,  etc). 

-  The  DoD  Sponsor  identify  current  Centers  of  Expertise  required 
for  improved  analyses. 


The  working  group  concluded  that  the  first  bullet  was  probably  the  most  critical 
follow-on  task  for  J8  and  ASD  (C3I)  to  tackle.  If  this  is  not  done  KM  will  fail. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  6  of  7) 


♦  Process  continued: 

-  Take  steps  to  systematize  the  organization  of  information  by 
transposing  principles  from  traditional  library  and  information 
sciences  to  the  electronic  realm. 

-  Develop  and  use  an  Information  Architecture  that  structures  the 
design  of  information  organization,  labeling,  navigation,  and 
indexing  systems  to  support  both  browsing  and  searching. 

-  Identify  and  use  a  set  of  metrics  for  the  business  case  preparation 
and  performance  evaluation  of  the  Analyst’s  Portal. 

-  Create  Knowledge-Sharing  Incentives  such  as: 

»  Knowledge-Sharing  inclusive  Performance  Reviews 
»  Embedded  Authorial  Recognition 

GB1  _ 


The  importance  of  performance  reviews  and  embedded  are  mentioned 
repeatedly  in  the  literature.  The  study  done  by  the  Working  Council  for  the 
Chief  Information  Officers  strongly  supports  this  point.  This  study  also  stressed 
the  importance  of  having  an  information  architecture  for  effective  design  and 
administration  of  browser-enabled  knowledge-based  repositories. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  7  of  7) 


♦  Technology 

-  The  DoD  concentrate  on  applying  COTS  tools  as  a  key  critical 
success  factor  for  building  analytical  KM  portal.  Example 
technologies  include: 

»  Thesaural  browsers 
»  Personalized  Intranet  Portal 

»  Knowledge-Sharing  Inclusive  Performance  Reviews 
»  Federated  Knowledge  Management 
»  Information-Expiration  Mechanisms 
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Fourteen  experienced  analysts  from  government,  the  FFRDCs,  and  industry 
participated  in  the  eight  working  group  sessions  and  should  be  given  the  credit 
for  the  contributions  I  will  present. 


For  our  purposes,  we  have  defined  collaborative  analysis  to  be  any  analytic 
activity  that  involves  the  sharing  of  data,  tools  and  expertise  across 
organizational  boundaries  to  achieve  a  common  goal. 
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WG  2  Presentations 


♦  COL  Bob  Clemence,  JCS/J-8,  Joint  Analysis:  A  Status  Report 

♦  LtCol  Kirk  Yost,  JCS/J-8,  Organizational  Alternatives  for 
Collaborative  Analysis 

♦  COL  Jim  Methered,  EUCOM,  The  Future  of  Joint  Analysis:  An 
Empirical  Look  for  Lessons 

♦  LTC  Jerry  Glasow,  DUSA(OR),  Input-Based  vs.  Output-Based 
Collaborative  Analysis 

♦  MAJ  Sue  Romans,  CAA,  Real  Time  Analysis 

♦  Ms.  Elaine  Simmons,  JDS,  Resolving  Data  Issues  in 
Collaborative  Analysis 

GUI 


We  used  contributed  papers  to  motivate  discussions  on  issues  that  individuals 
and  organizations  must  confront  before,  during  and  after  collaborative  work, 
such  as: 


•When  is  collaboration  useful? 

•How  should  collaborative  effort  be  organized? 

•If  information  is  power,  how  readily  will  organizations  share  their  data? 
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WG  #2  Enabling  Collaboration 
Focus  Questions 

♦  How  is  collaboration  currently  used  in  the  DoD  analysis 
community? 

♦  What  formal  and  informal  collaboration  techniques  can  be 
applied  to  improve  DoD  analysis? 

♦  What  analytic  practices  (local  study,  centrally-managed 
study,  etc.)  would  make  use  of  collaboration  support? 

♦  What  recommendations  do  you  have  to  improve  the  existing 
analytic  practice  using  collaboration? 


Four  focus  questions  were  proposed  before  the  Workshop  to  guide  discussions. 
In  our  deliberations,  we  could  not  make  a  distinction  between  questions  #1  and 
#3.  The  remainder  of  this  presentation  will  address  questions  #1,  2  and  4. 
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Current  Use  of  Collaboration  in  DoD  Analysis  Community 

♦  CINC  analyses  to  support  current  operations: 

-  Collaborative  by  nature  but  much  more  complex  than  in  past. 

-  National,  International  and  Joint  considerations. 

-  Compressed  decision  cycle  -  come  as  you  are  (HW,  SW,  data,  skill-set). 

-  Event  driven  and  spontaneous. 

-  Limited  by  lack  of  forward  analyst  presence  (few  collaborators  available). 

-  Frequent  requirement  for  Deployment  and  Logistical  Analysis. 

♦  DoD  level  analyses  to  answer  congressional  language  or  support 
PPBS: 

-  Collaboration  is  directed. 

-  Joint  and  Service  considerations  pre-dominate. 

-  Results  important,  but  not  urgent  -  time  to  acquire  tools,  data  and  skills. 

-  Formal  process  -  Terms  of  Reference,  Study  Plan,  Oversight. 
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We  have  identified  three  archetype  analytic  activities  where  collaborative 
analysis  takes  place. 

Analysis  in  the  unified  commands  is  collaborative  by  nature  due  to  the  military, 
political,  inter-agency,  and  international  considerations.  In  this  real-time 
environment,  the  decision-cycle  is  so  compressed  that  analysis  must  be  agile  to 
be  useful.  If  tools,  data,  and  talent  are  not  available  when  called,  the  decision 
will  be  made  without  analytical  insight.  Analysts  in  EUCOM  have  learned,  for 
example,  that  they  are  frequently  asked  to  provide  deployment  and  logistical 
analysis.  Collaborative  work  is  made  more  difficult  by  geographic  separation 
and  OPSEC  concerns. 

Within  the  Beltway,  collaborative  analysis  is  frequently  directed  to  answer 
Congressional  Language  or  to  support  PPBS  decisions.  In  this  environment, 
joint  and  service  considerations  pre-dominate.  Unlike  the  environment 
previously  described,  these  activities  are  important,  but  not  so  urgent  that  tools, 
data,  and  skills  cannot  be  massed  and  employed  in  a  deliberate,  formal  way.  A 
distinguishing  characteristic  of  this  kind  of  collaborative  work  in  a  hierarchical 
oversight  structure  that  guides  the  analysis  from  cradle  to  grave. 
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WG  #2  Enabling  Collaboration 

Current  Use  of  Collaboration  in  DoD  Analysis  Community 


♦  Service  analyses  to  support  acquisition,  PPBS  and  doctrine 
development: 

-  Collaborative  when  analysis  is  set  in  a  Joint  context. 

-  Service  considerations  predominate. 

-  Collaboration  may  be  formal  or  informal. 


The  third  collaborative  analysis  archetype  is  the  service/agency  study  performed 
to  meet  an  internal  requirement.  Intra-service  and  inter-service  collaboration  is 
driven  by  the  needs  of  the  service  customer.  Analyses  of  Alternatives  (AoAs), 
for  example,  for  Acquisition  Category  I  (ACAT-I)  systems  frequently  require  a 
joint  context.  Consultation  and  rendered  assistance  across  organizational 
boundaries  may  be  formal  (by  memorandum)  or  informal  (by  professional 
courtesy). 
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WG  #2  Enabling  Collaboration 

Formal  and  Informal  Techniques  That  Enable  Collaboration 

♦  Direct  Analytical  Support: 

-  Example:  Deployment  from  CAA  to  EUCOM  to  support  exercise. 

-  Not  responsive  enough  for  crisis  action  planning. 

-  Working  relationships  (organizational,  personal)  established  in 
advance  and  rehearsed. 

-  Co-location  is  preferred  to  tele-presence. 

♦  Data  Warehousing: 

-  Trusted  agent  for  data  verification,  validation  and  archival. 

-  Adherence  to  formal  data  access  and  release  procedures. 

♦  Trust-Building  Activities: 

-  Participation  in  MORS  working  groups,  workshops,  etc. 

-  Positive  experiences  with  earlier  collaborative  work. 

im 


Collaboration  can  be  encouraged  and  assisted  by  both  formal  and  informal 
mechanisms.  One  motivation  for  an  organization  to  collaborate  is  acquire  the 
use  of  tools  and  expertise  to  meet  infrequent  requirements  (e.g.,  support  to  Joint 
exercises  or  contingency  plan  development).  EUCOM  and  Center  for  Army 
Analysis  believe  that  this  arrangement  works  best  when  working  relationships 
have  been  established  in  advance  and  are  rehearsed.  In  this  way,  the  supporting 
organization  understands  the  decision  making  processes  of  the  supported 
organization  and  the  hardware/software/security  requirements  for 
interoperability.  While  collaborative  work  can  be  done  from  a  distance, 
EUCOM  prefers  on-site  support.  Co-location  helps  allay  fears  concerning 
operations  security,  responsiveness  and  miscommunication. 

The  existence  of  an  “honest  broker”  for  data  encourages  participants  to  share 
their  data  holdings.  The  Joint  Data  Support  (JDS)  Office  has  earned  the  trust  of 
the  Services,  the  Joint  Staff,  and  other  offices  in  the  Secretariat  by  the  way  it  has 
controlled  the  data  entrusted  to  it  during  Mobility  Requirements  Study  2005 
(MRS-05).  Perhaps  more  important  than  its  role  as  a  “gatekeeper”  is  the  service 
JDS  provides  to  the  community  by  cross-checking  the  data  it  receives  against 
other  authoritative  sources  to  identify  data  anomalies.  These  anomalies  are 
provided  back  to  the  source  for  resolution,  improving  the  veracity  and 
consistency  of  DoD  data  holdings. 

Informal  trust-building  activities  like  MORS  and  positive  experiences  with 
collaborative  analysis  help  to  overcome  organizational  and  individual  resistance 
to  teaming. 


WG2-7 


WG  #2  Enabling  Collaboration 

Observations  on  Collaborative  Analysis 


♦  Collaboration  only  works  when  people  and 

organizations  are  willing  to  set  aside  parochial  (or 
cultural)  behaviors  for  a  higher  purpose: 

-  Common  Goals  Do  Not  Guarantee  Consensus. 

-  Achieving  consensus  does  not  encourage  bold  shifts. 


♦  Paradigms  for  collaborative  studies  achieve  different 
outcomes 

centralized  (input-based)  vs  distributed  (output-based): 


Centralized  -  control  the  inputs  and  models  to  produce  one,  accredited 
result. 

»  Susceptible  to  single-point  failure;  deterministic  approaches  produce 
point  estimates. 

Distributed  -  encourage  independent  efforts  using  the  same  assumptions 
to  produce  a  distribution  of  results  for  executive  consideration. 

»  More  robust,  but  any  participant  can  warp  the  process  by  submitting 
biased  input. 

»  Requires  reconciliation  of  independent  results. 


It  is  important  to  remember  the  collaborative  analysis  is  not  an  appropriate 
technique  for  all  situations.  A  quote  attributed  to  Napoleon  is  “I  would  rather 
fight  allies  than  be  one.”  Sharing  common  ends  does  not  guarantee  agreement 
on  the  ways  and  means  used  to  get  there.  Group  pressure  to  cooperate  can  result 
in  compromises  that  preserve  the  status  quo  when  what  was  really  needed  was 
creativity  and  a  bold  shift. 

LTC  Jerry  Glasow  presented  a  paper  in  our  working  group  that  classifies 
collaborative  methods  as  being  either  “input-based”  or  “output-based.”  MRS- 
05  typifies  the  former  while  the  Phased  Threat  Distribution  (PTD)  Study  is  an 
example  of  the  later.  The  “input-based”  approach  reasons  that  if  the  participants 
accredit  (1)  the  data,  (2)  the  tools  and  (3)  the  analysts,  the  results  will,  in  turn, 
be  accredited.  When  this  approach  is  implemented  by  a  sequence  of  models 
(e.g.,  air  defense  precedes  deployment  precedes  force-on-force  evaluation)  a 
single  point  failure  can  delay  the  entire  project.  It  also  tends  to  produce  point 
estimates  rather  than  a  range  of  possible  outcomes  for  consideration. 

Two  advantages  of  the  “output-based”  approach  are  parallel,  independent 
analysis  and  multiple  results  that  can  be  compared.  The  most  difficult  part  of 
this  approach  can  be  determining  why  the  results  are  different.  If  this  approach 
is  used,  several  integrated  teams  should  be  commissioned  by  the  participants 
rather  than  putting  the  participants  in  competition  with  one  another. 
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Observations  on  Data  Collection  and  Production 

♦  Data  collection,  production  and  preservation  need  to  be  integrated 
into  day-to-day  operations: 

-  Need  to  address  multi  level  security  issues. 

-  Need  to  capture  operational  data  at  JTF  level  while  data  is  still  purple. 

-  Need  to  plan  for  data  capture. 

♦  Collaboration  slows  as  data  collection  and  production  burden 
increases: 

-  Deluge  of  requests  during  operations. 

-  OPSEC  issues. 

♦  Must  understand  limits  of  data: 

-  Close-hold  /  incomplete  data. 

-  Data  collection,  production,  preservation  are  manpower  limited. 

♦  Documentation  critical  to  accrediting  data  for  collaborative  use 

-  Includes  metadata  (technical,  informational,  operational). 


Data  collection,  production,  preservation  and  maintenance  was  a  recurring 
theme  in  our  deliberations.  Jim  Methered,  citing  his  experience  during  and  after 
OPERATION  ALLIED  FORCE,  is  vehement  that  data  collection  must  be  a 
component  of  the  JTF  information  management  plan  instead  of  the  “ad  hocery” 
that  exists  today: 

“...scavengers  picking  over  small,  rapidly  decaying,  incomplete,  and 
questionable  data  and  then  fiercely  defending  the  scraps  gathered  from  others.” 

A  potential  major  future  role  for  operations  analysis  may  be  to  study  how 
information  is  used  and  develop  ways  of  managing  and  using  it  better.  Process 
engineering  in  peace  might  facilitate  analysis  in  war  as  well  as  improve  actual 
and  simulated  decision  making  processes. 
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Recommendations 


♦  People 

-  Find  ways  to  identify  (and  evaluate?)  Subject  Matter  Experts  (SMEs). 

-  Find  a  means  to  develop  operational  experience  in  Civil  Service  1515s 
and  uniformed  analysts  w/o  warfighting  specialties. 

♦  Process 


There  is  a  spectrum  of  collaborative  analysis  processes,  and  each  has 
its  strengths  and  weaknesses  (bold  shifts  versus  consensus). 

»  Can  organize  for  competitive  analyses. 

»  Can  organize  to  search  for  a  “common  ground”  solution. 
Collaborative  Analysis  is  a  “teamsport”  and  requires  practice. 

»  OA  support  to  commanders  in  the  field  most  effective  when  “train 
as  you  fight”  has  been  implemented. 

Reestablish  formal  liaison/exchanges  to  improve  the  quality  of 
analysis. 


Our  recommendations  have  been  collected  under  the  headings  of  “People,” 
“Process”  and  “Product.” 


Collaboration  requires  colleagues  with  whom  to  collaborate.  We  believe  there 
should  be  a  better  way  of  identifying  the  experts  in  our  community  than  word  of 
mouth.  We  also  recognize  that  changes  in  the  way  the  uniformed  services  are 
managing  the  careers  of  military  analysts  may  make  collaborative  work  more 
essential  in  the  future.  We  must  either  find  a  way  to  increase  the  “field 
experience”  of  our  analysts  or  find  ways  to  include  “operators”  in  our  analysis 
teams. 


Collaborative  analysis  takes  longer  to  perform  than  analysis  done  in  isolation 
Timelines  can  be  shortened  if  informational  exchanges  and  liaisons  between 
analytic  organizations/agencies  are  institutionalized  in  our  business  practice. 
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Recommendations 

♦  Process  (Cont’d) 

-  Support  independent,  accredited  sources  for  data  to  speed  collaborative 
effort. 

-  Find  ways  to  identify  data  to  source  new  applications  and  phenomena. 

-  Data  collection  must  be  deliberate  and  done  in  such  a  way  that  it  does 
not  require  additional  effort  by  operators. 

-  Synchronize  data  production  with  data  demands  and  institutionalize 
production  and  update  of  long-lead  items. 

♦  E.g.,  extend  JMEM  to  include  developmental  systems. 

-  Harmonize  schedule  of  major  collaborative  projects  and  Service 
workload. 


mu 


In  the  short-term,  its  important  that  we  reconcile  our  demands  for  data  with  our 
ability  to  produce  it.  Dr.  Chris  Lamb,  ASD  (Strategy  and  Threat  Reduction)  is 
currently  leading  an  effort  with  the  services  and  the  intelligence  agencies  to 
better  coordinate  our  efforts.  It  also  makes  good  sense  to  plan  in  advance  for 
the  long-lead  data  that  are  common  to  many  of  the  studies  we  do  (e.g..  Time- 
phased  Force  Deployment  Data  or  TPFDDs)  In  the  long-term,  data  collection  in 
the  operational  environment  needs  to  evolve  from  something  that  interferes  with 
task  performance  to  something  that  is  transparent  to  people  performing  their 
day-to-day  tasks.  Similarly,  directives  to  perform  large,  collaborative  analysis 
needs  to  be  harmonized  with  the  day-to-day  analytic  workload. 
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Recommendations 


♦  Product 

-  Need  improvements  in  literature  search  capabilities,  identifying 
work  in  progress  that  may  affect  the  project  at  hand. 

-  Study  documentation  techniques  must  also  improve,  allowing  for 
reproducibility  of  results  and  reuse  of  data  in  addition  to 
description  of  procedure  and  findings. 


Compressed  decision-cycles  and  the  tendency  toward  using  briefings  rather  than 
reports  as  deliverables  has  lessened  our  use  of  literature  search  and  our 
discipline  to  thoroughly  document  what  we  do. 
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Working  Group  #3  Presentations 


♦  Dr.  Jackie  Henningsen,  Air  Force  Analysis  in  the  Context  of 
Today’s  Expeditionary  Aerospace  Force 

♦  Capt  Russ  Bergeman,  Analytic  Challenges  of  the  USMC 

♦  Mr.  Chuck  Werchado,  OPNAV  Naval  Analysis:  Recent  Navy  PPBS 
Process  Changes  and  QDR  Analysis  Initiatives 

♦  LTC  Jeff  Appleget,  Challenges  in  Army  Modeling 

♦  Mr.  Mike  McCurdy,  Analysis  Challenges  for  Joint  Operations  -  A 
USCINCPAC  Perspective 
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Applicable  Workshop  Goals  and  Objectives 


♦  Questions 

-  What  problems  are  not  well  served  by  the  current  state  of  the 
practice? 

♦  Goals 

-  Develop  a  shared  understanding  of  the  need  to  evolve  the  practice 
of  Military  Operations  Analysis  in  DoD. 

-  Establish  a  vision  for  improving  analytic  practice. 

-  Objectives. 

-  Identify  weaknesses  in  the  current  practice  relative  to  current  and 
future  requirements. 


The  objective  of  Working  Group  #3,  New  Analytical  Challenges,  was  to  address 
the  weaknesses  in  the  current  practice  of  analysis  within  the  Department  of 
Defense. 

The  working  group  focused  on  giving  each  of  the  services  and  the  Joint  Staff  the 
opportunity  of  presenting  their  current  analytical  challenges... problems  that 
they  are  facing  that  cannot  be  solved  with  the  current  analytical  toolset.  The 
working  group’s  goal  was  to  define  the  areas  that  need  to  be  evolved  within  the 
practice  of  DoD  analysis. 
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Focus  Question 


♦  What  are  military  operations  analysis  questions  of  interest  for 
which  the  established  military  operations  analysis  community 
apparently  does  not  have  answers? 

-  i.e..  What  problems  are  not  well  served  by  the  current  state  of  the 
practice? 


EBB 


The  focus  of  Working  Group  #3  was  to  examine  military  operations  analysis 
questions  of  interest  for  which  the  established  military  operations  analysis 
community  apparently  does  not  have  the  answers.  This  working  group  is  about 
new  problems,  problems  not  involving  conventional  warfare,  and/or  post-Cold 
War  problems.  Actually  there  are  old  problems  that  belong  here  as  well.  The 
issue  isn’t  whether  the  problem  is  new  or  old,  but  rather  whether  the  Military  OR 
community  has  succeeded  in  addressing  it  satisfactorily.  An  example  is  the 
problem  of  determining  the  value  of  a  common  operational  picture  to  the 
warfighter.  Presentations  to  this  working  group  came  from  the  major  analysis 
players  such  as  the  Services’  Studies  and  Analysis  shops,  from  warfighters  like 
the  Joint  Forces  Command  and  USCINCPAC,  and  from  decision  makers  like 
OSD,  J8,  and  the  Services'  warfare  requirements  and  assessment  organizations. 
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Problems  not  well  served  by  the  current  state  of  MOR  practice 


♦  Methods  for  Analyzing  New  Domains  of  Operation 

-  High  intensity/low  casualty  conflict 

-  MOOTW,  humanitarian  operations,  peacetime  engagement, 
operations  in  urban  terrain 

-  Technology  insertion  (digital  battlefield,  tomorrow’s  dismounted 
warrior) 

-  Chemical/biological  warfare  effects  (including  contagion) 

-  Asymmetric  threats  (terrorism,  political  manipulation,  homeland 
defense) 

»  Means  for  recognizing  attack 

-  Application  and  effects  of  special  operations 

-  Information  operations/warfare 

B3 


With  the  end  of  the  Cold  War,  the  Department  of  Defense  and  thus  the  DoD 
analytical  community,  are  faced  with  the  demand  of  conducting  and  analyzing  a 
variety  of  new  operations.  These  new  operations  require  new  methods  of 
analysis  for  planning,  conceiving  concepts  of  operations,  defining  doctrine  and 
analyzing  effectiveness.  Some  examples  of  these  new  domains  are:  High 
Intensity/Low  Casualty  Conflict,  Military  Operations  Other  Than  War 
(MOOTW),  humanitarian  operations,  peacetime  engagement,  operations  in 
urban  terrain  technology  insertion  (digital  battlefield,  tomorrow’s  dismounted 
warrior),  chemical/biological  warfare  (including  contagion),  asymmetric  threats 
(domestic  terrorism,  political  manipulation,  homeland  defense),  special 
operations  and  information  operations/warfare. 


WG3  -6 


WG  #3  New  Analysis  Challenges 

Problems  not  well  served  by  the  current  state  of  MOR  practice 


♦  Model-centric  focus  limiting  analysis  toolkit 

-  Scenario  development  and  data  collection  time  consuming 

-  More  scoping  analysis  to  set  stage  for  detailed  study 

-  Encourage  exchange  of  simple  analysis  tools  (e.g.,  spreadsheets) 

-  Combine  results  from  all  pertinent  tools 

♦  Identifying  complex  interrelationships  between  factors 

-  communicate  effects  of  multi-dimensional  trade  space 


The  current  emphasis  on  modeling  and  simulation  with  it’s  resulting  de¬ 
emphasis  on  other  analytical  tools  and  methods  is  limiting  the  analysis  toolkit. 
With  the  amount  of  time  and  resources  required  for  scenario  development  and 
data  collection  to  run  the  current  models  and  simulations,  some  other 
alternatives  are  clearly  needed.  For  example,  methods  of  scoping  analysis  that 
can  quickly  look  at  the  alternatives  and  potentially  set  the  stage  for  further, 
larger,  study  are  needed.  These  scoping  tools  can  serve  to  pare  down  the 
alternatives  and  allow  the  focus  on  the  important  question  to  evolve.  In 
addition,  simpler  analysis  tools  such  as  spreadsheets  should  be  used  and 
exchanged  within  the  analytical  community  as  a  rapid  means  of  knowledge 
transfer.  Finally  the  analyst  should  be  taught  to  combine  the  results  from  all 
pertinent  tools  and  analytical  methods  to  more  thoroughly,  accurately  and 
quickly  answer  the  question  at  hand. 

With  the  increasing  complexity  of  today’s  military  scenarios  and  high 
technology  weapons,  there  is  a  greater  need  for  the  analysis  community  to  be 
able  to  identify  and  communicate  the  effects  of  the  factors  and  their 
interrelationships.  It  is  especially  important  to  be  able  to  conduct  and 
communicate  the  trades  conducted  between  factors  that  typically  would  not  be 
linked.  For  example,  an  aircraft  earner  can  be  the  platform  used  for  JSF 
launches,  and  a  typical  analysis  may  look  at  the  number  of  targets  put  at  risk  by 
various  sizes  of  JSF  fleets.  However,  an  interrelated  measure  that  should  be 
considered  is  the  deck  space  that  is  being  taken  up  by  the  JSF  fleet,  that  may 
potentially  be  used  for  such  things  as  helicopter  medevac  support. .  .two  very 
important  missions  with  interrelations  that  must  be  considered. 
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Problems  not  well  served  by  the  current  state  of  MOR  practice 


♦  Analytical  methods  that  trace  factors  to  operational  and 
strategic  outcomes: 

-  Logistics 

-  Bandwidth 

-  Engagement/presence  on  PERSTEMPO  and  OPTEMPO 

-  Infrastructure  Investments 

-  Readiness  postures 

-  Force  structure  changes  (deployable,  lighter,  lethal  fighting  force) 


There  are  many  factors  that  effect  the  outcome  of  a  battle,  but  do  not  have 
directly  measurable  combat  effects.  These  factors  must  be  taken  into  account, 
and  their  effects  must  be  traced  to  operational  and  strategic  outcomes.  These 
factors  include  such  things  as  logistics,  bandwidth  requirements, 
engagement/presence  on  PERSTEMPO  and  OPTEMPO,  infrastructure 
investments,  readiness  postures  and  various  force  structure  alternatives. 
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Problems  not  well  served  by  the  current  state  of  MOR  practice 


♦  Methods  for  including  behavior  factors  in  current  tools 

-  Quantifying  small  unit  behavior 

»  Effect  of  situational  awareness  on  force  effectiveness 

-  Quantifying  adversary  leadership  motivation 

»  Effect  of  target  selection  in  influencing  course  of  action 

♦  Methods  of  quantifying  risk 

♦  Real  time  analysis  for  operational  support 


The  behavior  of  individuals,  small  units,  and  adversary  leaders  must  be  captured 
in  our  evolving  analytical  tools.  The  effect  and  measurement  of  behavior 
factors  at  the  individual  or  small  unit  levels  are  important  considerations, 
especially  in  the  analysis  and  evaluation  of  combat  systems.  For  example,  the 
increase  in  situational  awareness  for  the  individual  foot  soldier  is  probably  a 
good  thing,  to  a  point  —  until  information  overload  occurs  or  if  the  equipment 
used  to  give  the  situational  awareness  decreases  the  individual’s  effectiveness  in 
other  areas,  such  as  mobility.  On  the  other  hand,  being  able  to  quantify  the 
characteristics  and  motivations  of  an  adversary  leader  would  allow  the 
identification  of  the  leader’s  or  country’s  centers  of  gravity  and  may  assist  the 
planning  process  in  identifying  effective  target  sets  that  would  influence  the 
enemy’s  course  of  action. 

Methods  for  quantifying  risks  of  all  sorts  must  be  developed  and 
institutionalized. 

The  frequency  of  calls  for  quick-tum  around  analysis  is  increasing.  These 
analyses  take  many  forms,  such  as  support  for  testimony  to  Congress  or  support 
to  ongoing  warfighting  efforts.  To  answer  these  types  of  questions,  the  analyst 
needs  a  set  of  tools  that  support  these  quick  analysis  activities.  These  tools  need 
to  capture  the  important  elements  that  would  be  drawn  out  by  more  thorough 
analysis  but  run  quickly  enough  to  support  answering  the  question  when  the 
answer  will  still  be  desired  and  relevant. 
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Recommendations 


♦  Find  methods  to  deal  with: 

-  New  forms  of  operations. 

-  Traceability  of  known  factors  and  interactions. 

-  Quick  turn-around  studies. 

♦  Nurture  analysts  and  analytical  thinking  rather  than  focusing  on 
model  development: 

-  Project  more  balanced  view  of  model  capabilities  and  limitations. 

-  Reverse  trend  to  big  models  as  solution  to  all  problems. 

♦  Characterize  and  communicate  impact  of  uncertainties  (e.G., 
Risk,  behavioral  factors). 

(US 


In  summary,  as  the  analytical  community  evolves  it  needs  to  find  tools  to  deal 
with  new  forms  of  operations,  find  methods  to  trace  known  factors  and 
interactions  to  their  logical  combat  output,  and  find  tools  that  will  provide 
accurate  and  thorough  analysis  in  a  short  timeline. 

The  DoD  analytical  community  needs  to  nurture  analysts  and  emphasize 
analytical  thinking  rather  than  focusing  on  simulation  and  modeling 
development.  The  community  must  project  a  more  balanced  view  of  model 
capabilities  as  well  as  limitations,  and  reverse  the  recent  trend  of  blind  faith 
toward  large  models  as  the  solution  to  all  analytical  problems. 

The  analytical  community  needs  to  find  means  for  characterizing  and 
communicating  the  impacts  of  uncertainty.  For  example  risk,  such  as  decision 
risk  based  on  the  fact  that  various  factors  may  not  have  been  considered  in  the 
current  analysis,  need  to  be  characterized  and  communicated  to  the  decision 
maker. 
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Backup  Slides  —  Session  Notes 


The  following  slides  provide  the  notes  taken  during  the  presentations  and  the 
resulting  discussions  of  the  working  group  sessions.  The  problems  identified  in 
the  previous  slides  were  distilled  from  these  session  notes. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Methods  for  recognizing/analyzing  unrestricted  warfare 

-  New  domains  of  warfare,  warfare  reborn 

-  Terrorism,  high  intensity/low  casualty  conflict 

♦  Apply  traditional  OR  methodologies,  but  consider  methods  for 
including  behavior  factors 

-  Draw  upon  expertise  of  multi-disciplinary  teams  (behavioral 
scientists,  psychologists,  etc.) 

-  Effect  of  target  selection  to  force  enemy  capitulation 

♦  Need  measures  to  relate  warfare  outcome  of  non-warfare  factors 
and  metrics  for  other  operations 

-  Logistics,  bandwidth 

-  MOOTW,  humanitarian  operations 


During  her  keynote  address,  Dr.  Ruth  Davis  introduced  the  concept  of 
unrestricted  warfare.  With  this  concept,  war  has  been  reborn,  focused  on  such 
activities  as  terrorism  and  high  intensity/low  casualty  conflict.  These  types  of 
warfare  emphasize  the  need  to  analyze  economic  and  police  efforts,  something 
that  the  current  analytical  community  tools  are  ill  prepared  to  handle. 

We  need  to  recognize  that  the  behavioral  science  fields,  although  not  as 
measurable  as  other  tools,  provide  useful  information  to  policy  decision  makers. 
Part  of  OR  analysis’  value  to  decision  makers  may  be  to  use  these  sciences  to 
identify  alternatives  or  structure  the  decisions.  For  example,  developing  a 
model  that  relates  an  adversary  leaders’  motivations  and  vulnerabilities,  could 
potentially  be  used  to  identify  the  target  set  that  would  force  that  leader  into 
capitulation. 

The  use  of  measures  that  quantify  warfare  outcomes,  such  as  attrition,  are  well 
known.  However,  there  are  many  factors  that  support  warfare  operations  that 
are  not  directly  quantifiable  with  the  current  measures  and  analytical  techniques 
The  community  needs  to  identify  metrics  that  can  be  used  to  quantify  the  value 
of  these  non-warfare  factors,  such  as  logistics  and  bandwidth  use  and  other 
types  of  operations,  such  as  MOOTW  and  humanitarian  operations. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Communicate  effects  of  multi-dimensional  trade  space: 

-  Different  system  solution  options,  multi  service. 

♦  With  changing  CONOPS  and  doctrine,  analysis  methods  and 
human  factors  assessments  must  catch  up: 

-  Information  technology  insertion  (digital  battlefield,  tomorrow’s 
dismounted  warrior). 

-  Force  structure  changes  (deployable,  lighter,  lethal  fighting  force). 

-  Operation  in  urban  terrain. 


The  analytical  community  needs  to  find  new  techniques  that  will  allow  the 
capability  to  communicate  the  effect  of  the  factors  within  a  multi-dimensional 
trade  space.  This  trade  space  includes  trading  various  system  solutions  or 
options  offered  across  service  boundaries. 

The  new  environment  that  includes  rapidly  advancing  technology  and  the 
paradigm  that  “change  is  good”  is  rapidly  changing  CONcepts  of  OPerationS 
(CONOPS)  and  doctrine.  While  the  way  the  military  does  business  is  rapidly 
changing,  the  environment  is  also  providing  new  analytical  challenges.  We 
need  to  figure  out  how  we  fit  into  this  new  environment,  and  develop  tools  that 
will  answer  questions  such  as: 

-  How  to  assess  the  benefits  and  drawbacks  of  information  technology  insertion, 
such  as  the  digital  battlefield  and  tomorrow’s  dismounted  warrior.  For  example, 
is  there  a  point  where  too  much  information  is  too  much  for  the  warrior? 

-  How  to  assess  the  benefits  and  implications  of  proposed  force  structure 
changes. 

-  How  to  model  operations  in  urban  terrain,  such  as  determining  the  level  of 
detail  of  modeling  buildings  in  an  urban  environment. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Challenges  from  analytical  support  to  CINC’s 

-  Analysis  tools  and  data  for  OOTW  and  peacetime  engagement 

-  Indicators  of  inter/intra-state  instability 

-  Real-time  analytical  support  for  operations  and  exercises 

»  Short  analysis  time  frame 

-  Field  analysis  of  operations  and  exercises 

-  Combining  training  and  analysis 

»  Simulation  support  for  mega-CAX’s 
»  Combining  analysis  and  exercise  results 


dm 


From  the  CINC’s  viewpoint,  there  are  a  variety  of  analysis  challenges. 

Real  time  analytical  support  for  operations  offers  a  different  environment,  due 
to  the  very  short  timeframe  allowed  for  the  analytical  activities.  There  is  no 
time  to  build  data  bases,  therefore  the  most  valuable  resource  that  the  analyst 
can  offer  is  an  operational  focus  and  a  structure  for  thinking  about  the  problem. 

Training  simulations  provide  challenges  in  a  few  areas.  Increasing  the 
simulation  support  for  combining  many  of  the  large  exercises,  and  combining 
analysis  and  exercise  results. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 

♦  QDR  analysis  challenges: 

-  Methods  of  quantifying  risk  and  uncertainty. 

-  Feedback  loops  to  include  analysis  in  planning. 

-  Analytical  transparency. 

-  Effects  quantification: 

»  Engagement/presence  on  PERSTEMPO  and  OPSTEMPO. 

»  Infrastructure  Investments. 

»  Readiness  Postures. 

»  Chemical/Biological  Warfare  Effects. 

»  Asymmetric  Threats  (terrorism,  political  manipulation). 

»  Information  Operations AVarf are. 

-  Compares  Four  Alternative  Force  Structure  Emphases  against  Four 
Scenarios. 


These  analysis  challenges  were  extracted  from  Tom  Allen’s  QDR  Workshop 
presentation  during  Wednesday’s  plenary  session. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Navy  Analytical  Challenges: 

-  Littoral  Challenge. .  .Post  Cold  War  Emphasis 

-  Capabilities-Based  Analysis  for  Resource  Decisions 

»  How  to  Map  Support  Missions  to  Warfighting  Capabilities 

-  Model  Available  to  Define  Capability  to  Meet  Mission 
Requirements  from  Home  Port  to  Crisis,  However... 

»  MOOTW  Analysis  Test  Cases  Varied  and  Changing 

♦  Need  Stability  of  Dynamic  Commitment  Scenarios 

-  Fleet  battle  experiment  challenges 

»  Data  collection  and  preservation 
»  Prioritization  among  real  world  operations,  training,  etc. 

»  What  are  the  measures  of  success  for  FBE? 

♦  Innovation,  translation  to  operations? 

dS3 


Without  the  existence  of  a  comparable  Navy  opponent  for  battles  on  the  high- 
seas,  the  Navy  will  be  faced  with  a  “new”  form  of  warfare. .  .fights  in  the 
littorals.  This  new  form  of  warfare  will  require  new  analytical  tools  to  help 
define  the  appropriate  force  structures,  doctrine,  and  tactics  required  to  fight 
these  crises. 

The  IWAR/CPAM  system  is  an  end-to-end  capabilities  analysis  method  used  to 
make  resource  decisions.  The  analytical  challenge  provided  by  this  is  to  define 
the  linkage  between  the  support  missions  to  their  contribution  to  warfighting 
capabilities. 

The  IWAR/CPAM  process  uses  a  model  (FORSAT)  that  defines  the  capability 
to  meet  mission  requirements  from  home  port  to  the  crisis  area.  The  issue 
provided  with  this  tool  is  the  definition  of  the  MOOTW  scenarios.  The 
scenarios  are  varied,  changing,  and  growing  by  the  day.  The  analysts,  with 
current  funding  levels,  cannot  keep  up  with  the  development  of  the  scenarios 
needed  to  describe  the  MOOTW  conditions. 

The  Fleet  Battle  Experimentation  (FBE)  program  provides  the  Navy  a  means  for 
evaluating  new  operational  concepts  within  the  fleet  environment.  Analytical 
challenges  presented  by  the  FBE  is  data  collection  and  preservation,  and  thus 
translation  into  doctrine  and  operations.  Due  to  the  conduct  of  FBE  within  the 
active  fleet,  it  necessarily  takes  a  back  seat  to  real  world  operations  and  other 
priorities  during  fleet  operations. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Air  force  analytical  challenges 

-  Expeditionary  aerospace  force  focus 

»  Vision,  guidance,  effects  based  operations  with  consideration 
of  future  warfare  and  risks  guide  modernization  planning 

♦  Metrics  to  evaluate  effects  based  operations 

♦  Detailed  analysis  needed  to  guide  the  transformation  and 
innovation 

-  Integration  of  analysis  activities 

»  Experimentation,  wargames,  battlelabs,  ACTDs 


A  previous  CSAF  decided  that  analysis  did  not  fit  well  with  his  vision  of  the  Air 
Force.  He  made  the  decision  to  decrease  the  size  and  organizational  standing  of 
the  Air  Force  Studies  and  Analysis  Agency  (AFSAA).  This  decline  has  had 
many  negative  aspects,  including  the  separation  of  blue  suit  analyst  from 
contractor  supported  analysis.  In  many  cases,  the  linkage  between  the  analysis 
and  decision  making  has  been  lost.  At  the  same  time  that  the  analysis  capability 
has  been  drawn  down  and  de-emphasized,  there  has  been  an  increase  in  analysis 
“911”  calls  from  various  sources  such  as  CSAF,  Congress,  AF/XP,  etc. 

The  Expeditionary  Aerospace  Force  concept  offers  many  challenges.  The 
combination  of  vision,  guidance,  and  the  consideration  of  effects  based 
operations  combine  to  guide  modernization  planning.  During  this  planning 
process,  future  warfare  and  the  decision  risks  involved  must  be  considered. 
These  radical  changes  in  the  AF  doctrine  and  operations  must  be  guided  by 
thorough  analysis  activities  that  will  guide  the  transformation  and  innovation. 

The  need  to  systematically  link  all  analysis  activities  is  desperately  needed  to 
include  the  information  and  knowledge  gained  from  the  various  activities,  such 
as  experimentation,  wargaming,  battlelab  initiatives  and  ACTD’s  /ATD’s 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Air  Force  analytical  challenges  (continued): 

-  Model-centric  focus  limiting  analysis  toolkit 

»  Scenario  development  and  data  collect  time  consuming. 

»  Need  “scoping  models”  to  quickly  assess  alternatives. 

-  Global  vs.  local  optimization 

»  OR  community  has  the  training  to  understand  both. 

»  Need  tools  to  combine  local  (service  specific)  suggestions  into 
global  (Joint)  solution. 

-  Technology  balance 

»  Need  to  be  careful  of  chasing  technology  at  the  expense  of 
intellect. 


The  current  emphasis  on  M&S  as  the  primary  analysis  tool,  limits  the  capability 
of  the  analyst.  Scenario  development  and  data  collection  required  for  high- 
resolution  models  are  extremely  time-consuming  tasks  and  thus  limit  the 
analyst’s  ability  to  support  quick-tumaround  or  small  analysis  questions.  A  gap- 
filler  or  “scoping  model”  is  required  to  quickly  assess  alternatives  that  may 
subsequently  be  studied  further  with  the  high  resolution  or  campaign  models. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Air  force  analytical  challenges  (continued) 

-  Joint  collaborative  analysis  NOT  joint  collaborative  decision  making 

-  Customer  for  all  ages 

»  Decision  maker  of  the  future  may  be  true  customer 
»  Warfighter  or  decision  maker  as  customer? 

-  Impact  of  logistics  on  successful  operations 

»  How  to  conduct  coshfaenefit  analysis  on  logistics  improvement 
investments 

-  Logistics  in  wargames  and  staff  training  exercises 

»  How  to  raise  awareness  of  the  importance  of  logistics 

-  Impact  of  IW  on  logistics 

-  Civilian  on  the  battlefield 

»  Impact  of  civilian  non-participation  once  battle  begins 

SB 


The  customer  for  our  analysis  capabilities  may  very  well  be  the  decision  maker 
of  the  future,  rather  than  current  decision  makers.  Given  this  focus,  how  do  you 
structure  the  analytical  community’s  activities  to  meet  the  needs  of  this 
customer. 

The  impact  of  logistics  on  the  battlefield  has  not  been  adequately  modeled  in  the 
current  modeling  tools.  For  example,  modeling  the  impact  of  logistics  on 
successful  operations  requires  the  need  to  do  things  such  as  cost/benefit  analyses 
on  logistics  improvement  investments,  and  quantifying  the  costs  and  benefits 
with  respect  to  combat  operations. 

In  order  to  raise  the  awareness  of  the  importance  of  logistics  in  senior  leadership 
and  planning  efforts,  there  is  a  need  to  include  logistics  considerations  in 
wargames  and  staff  training  exercises.  Rather  than  allowing  the  decision  maker 
to  assume  away  all  the  logistics  activities,  they  must  be  included  in  their 
thinking  processes. 

What  is  the  impact  of  information  warfare  on  the  conduct  of  logistics  and  thus 
the  conduct  of  combat  operations?  For  example  what  is  the  impact  on  combat 
operations  when  the  logistics  system  supported  them  is  attacked  and  brought 
down  by  information  warfare  operations? 

The  impact  of  civilian  participation  on  the  battlefield  needs  to  be  considered  and 
what  the  impact  of  their  loss  will  be.  For  example,  when  the  shooting  began 
during  Operation  Desert  Storm  the  UPS  pilots  providing  logistical  support 
refused  to  participate. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Marine  Corps  analytical  challenges 

-  Analytical  Resources 

»  Number  of  analyst  driven  by  manning  requirements  NOT 
analytical  requirements 

»  Less  analysts  and  more  complex  environment  producing 
increased  analyst  workload 

-  Identifying  interrelationships  between  factors 

»  JSF  strikes  vs.  helicopter  medevac  during  carrier  operations 

-  Increased  expectations  for  operations  analysis 

»  More  analytically  savvy  leaders  expecting  more  and  better 
products 

»  Subordinate  officers  have  unrealistic  expectations 

♦  M&S  is  panacea 

♦  Every  situation  requires  a  study 


The  USMC  is  facing  several  analytical  challenges:  conflict  of  analytical 
resources  and  requirements,  increased  expectations  of  analysis;  both  leading  to 
the  need  for  a  comprehensive  analysis  for  the  future. 

The  number  of  USMC  analysts  is  driven  by  manning  requirements,  not  by  the 
number  of  analytical  requirements  that  the  USMC  is  facing.  This  combination 
of  more  complex  analytical  problems  and  a  small  number  of  analyst  combine  to 
create  a  greater  workload  for  each  USMC  analyst. 

The  inability  to  define  and  analyze  the  interrelationships  between  systems, 
missions  and  functions  provides  an  analytical  challenge.  For  example, 
increases  in  JSF  sortie  generation  rate  allows  the  USMC  to  strike  more  targets 
ashore,  but  at  the  same  time  consumes  deck  space  that  may  have  been  used  for 
helicopters  that  could  evacuate  wounded  from  the  ship.  How  are  diverse,  yet 
interrelated  factors  traded  off  in  a  comprehensive  analysis? 

Increased  expectations  of  the  analytical  community  are  spawned  from  savvy 
senior  leaders  who  are  demanding  more  and  better  analytical  products.  On  the 
other  hand,  not  so  savvy  mid-level  leaders  introduce  unrealistic  expectations  for 
the  community.  They  bring  expectations  that  M&S  is  a  panacea  and  that  every 
question  needs  an  analysis. 
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WG  #3  New  Analysis  Challenges 

Session  Summary 


♦  Must  consider/use  other  analytical  tools 

-  Not  just  M&S 

♦  Analysis  plan  for  the  future 

-  Focus  on  Operational  Maneuver  From  The  Sea  (OMFTS) 

-  Event  vs.  time  driven  analysis 

»  Must  incorporate  lessons  learned  into  analysis  planning,  not 
just  blind  adherence  to  early  planning 


The  analytical  community  MUST  consider  other  tools  rather  than  just  focusing 
on  M&S  to  answer  every  question. 

Analysis  plan  for  the  future  focuses  on  Operational  Maneuver  From  the  Sea 
(OMFTS).  Lessons  learned  must  be  incorporated  in  the  planning  and  execution 
of  future  analytical  activities. 
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MORS  Workshop 

Evolving  the  Practice  of  Military 
Operations  Analysis  in  DoD 

Working  Group  #4 
New  Analysis  Tools  and  Methods 

Chair:  LTC  Dan  Maxwell 
Co-Chairs:  John  Furman  and  Dr.  Al  Brandstein 


February  29,  2000  through  March  2,  2000 
Naval  Postgraduate  School,  Monterey,  CA 


The  Charter  of  the  New  Analysis  Tools  Working  Group  was  to  identify 
emerging  tools  and  techniques  for  conducting  operations  analysis,  given  the  new 
challenges  that  face  the  military  operations  research  community  in  the  21st 
century. 

To  accomplish  this,  MORS  leadership  assembled  representatives  from  domains 
that  are  not  normally  part  of  the  traditional  MORS  population.  Experts  with 
expertise  in  C4I  systems,  intelligence,  policy  analysis,  artificial  intelligence  and 
knowledge  management  were  present.  This  eclectic  set  of  skills  resulted  in 
information  exchange  that  exceeded  expectations  —  a  vision  focused  on 
providing  a  foundation  of  consistent  and  compatible  data,  as  well  as  an 
architecture  that  could  provide  connectivity  between  command  and  control 
systems  and  analytic  modeling  environments. 

The  kinds  of  connectivity  and  consistency  that  are  envisioned  in  this  product  are 
believed  to  be  absolutely  essential  for  providing  commanders  and  staffs  in  the 
field  with  flexible  automated  planning  support,  and  keeping  analysis  relevant  in 
the  highly  uncertain  future  that  the  DoD  faces. 

Perhaps  most  importantly,  it  was  concluded  that  for  the  most  part  there  is  no 
technological  impediment  to  achieving  the  vision  that  is  articulated  herein,  and 
that  these  kinds  of  efforts  are  an  essential  part  of  achieving  the  knowledge 
management  “culture”  that  has  been  implemented  successfully  in  many 
commercial  enterprises. 
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Working  Group  #4  Presentations 


♦  MAJ  Curt  Doescher,  Modeling  and  Simulation  Standards 
Development 

♦  Dr.  David  Homer,  A  Standards-Based  Methodology  for 
Throughput  for  Use  in  Movement  Representation  in  Modeling  and 
Simulation 

♦  Ms.  Kay  Burnett,  Conceptual  Modeling  of  Foreign  Command 
Decision  Processes 

♦  Dr.  Mike  Hieb,  Data  Alignment  -  C4I  and  M&S 

♦  Ms.  Elaine  Simmons,  Employing  Data  Warehousing  Technologies 
in  Support  of  Analysis 

♦  CPT  John  McKitrick,  Army  Flow  Model 
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WG  #4  New  Analysis  Tools  and  Methods 
Focus  Questions 

♦  What  currently  emerging  developments  in  analytical  approaches  to 
complex  systems  can  be  applied  to  challenging  military  operations 
analysis  problems? 

-  Consider  areas  such  as  chaos,  complexity  theory,  etc. 

♦  To  what  DoD  analyses,  particularly  the  new  analytic  challenges 
defined  by  WG3,  can  these  approaches  be  applied? 

♦  What  recommendations  do  you  have  to  improve  the  existing 
analytic  practice  in  the  near  term  by  using  new  analysis  tools  and 
methods? 

♦  What  recommendations  do  you  have  for  areas  of  research  for  the 
next  generation  military  operations  analysis? 


The  MORS  leadership  asked  the  working  group  to  identify  developments  in 
analytical  approaches  to  complex  systems  that  were  emerging.  While  some  of 
the  areas  were  explored  in  the  course  of  discussion,  it  is  clear  that  there  was 
much  relevant  research  that  was  not  able  to  be  addressed  in  the  compressed  time 
schedule  of  a  three  day  workshop. 

We  attempted  to  map  our  discussion  to  the  products  of  working  group  three. 
However,  this  topic  is  better  addressed  by  the  report  of  the  synthesis  group. 

The  group  did  deliberate  and  prepare  a  set  of  recommendations  on  actions  that 
could  be  taken  by  the  DoD  to  improve  the  analytic  practice,  as  well  as  improve 
the  coordination  with  other  domains  of  expertise. 
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Objective 


♦  To  identify  emerging  techniques  and  tools  that  will  support 
military  operations  research. 

♦  To  map  these  concepts  to  the  Department’s  analytic  challenges 

♦  To  develop  a  set  of  near  term  investment  recommendations  for 
the  MORS  Sponsors’  consideration 

♦  To  develop  a  set  of  longer-term  research  and  development 
recommendations  for  the  MORS  Sponsors’  consideration 


Our  objectives  were  derived  directly  from  the  focus  questions  that  were 
presented  to  the  group  by  the  MORS  leadership. 

One  of  the  challenges  for  this  working  group  was  to  identify  real  needs  for 
improvements  in  tool  and/or  data  capabilities  given  knowledge  management’s 
emphasis  on  better  exploiting  existing  technical  capabilities.  We  searched  for 
existing  capabilities  to  exploit  and  attempted  to  organize  our  recommendations 
based  on  the  vision  you  will  see  in  a  later  slide. 
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Assumptions  and  Scope  of  Exploration 


♦  Knowledge  management  technologies  will  provide  a 
“repository”  in  which  these  tools  and  techniques  will  be  made 
available  to  analysts.  (Limit  WG  scope  to  models,  data  and  data 
development.) 

♦  Earlier  objectives  in  the  modeling  and  simulation  master  plan 
regarding  reuse  in  modeling,  simulation  and  analysis  still  hold. 

♦  The  hardware  and  software  infrastructure  will  support 
interoperability  of  systems  (e.g.  HLA  and  DIICOE). 

♦  Un-addressed  deficiencies  identified  in  the  analysis  functional 
working  group  report  remain  valid. 


Because  the  solution  space  was  potentially  vast  and  unmanageable  in  a 
workshop  setting,  the  group  made  initial  assumptions  to  bound  the  discussion. 

In  some  cases  the  assumptions  were  found  to  be  to  restrictive  and  were 
necessarily  relaxed. 

The  first  assumption  is  that  KM  experts  will  create  a  repository  infrastructure, 
and  that  infrastructure  will  be  available  for  use  to  store  and  distribute  analysis 
domain  knowledge. 

Analysts  and  the  DoD  have  been  modeling  and  simulating  for  a  long  time. 

There  is  an  existing  set  of  recommendations  in  the  DoD  Modeling  and 
Simulation  Master  Plan.  These  recommendations  are  assumed  to  be  good,  and 
future  recommendations  should  deviate  only  when  needed. 

There  are  known  architectures  available  to  us  to  move  information  between 
analytical  models  that  could  facilitate  a  knowledge  management  oriented  system 
and  culture.  One  example  is  the  High  Level  Architecture  (HLA).  There  are  also 
known  architectures  to  move  information  between  near  Real-Time  C4I  systems 
such  as  the  Defense  Information  Infrastructure  Common  Operating 
Environment.  While  it  is  logical  to  assume  that  these  architectures  will  ensure 
interoperability,  discussion  with  experts  from  the  C4I  domain  revealed  that  the 
assumption  does  not  currently  hold  with  respect  interoperability  between 
analytical  models  and  C4I  Systems  (i.e.,  the  HLA  and  the  DII  COE). 

The  Analysis  Functional  Working  Group  has  a  valid  list  of  analysis  deficiencies 
that  needs  to  also  be  considered.  We  anchored  on  how  to  do  big  studies  better 
then  started  to  look  at  minute-long  decision  cycles  (from  the  analysis  working 
group)  to  balance-out  this  discussion. 
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Assertions 


♦  Although  warfare  doctrine  and  military  missions  are  evolving, 
there  are  constant/recurring  analytic  requirements  that  can  and 
should  be  planned  for: 

-  Force  assessment 

-  Planning  and  execution 

»  Deliberate  planning 
»  Crisis  action  planning 

-  System  effectiveness  and  trade-off  analysis 

-  Concept  and  doctrine  development  and  assessment 


These  missions  are  fairly  constant  in  the  workspace  of  military  operations 
research.  There  is  ever  reason  to  believe  they  will  continue  to  be  the  main 
missions  of  the  future.  They  form  the  space  around  which  to  structure  the  new 
tools  for  analysis  challenges. 

Because  the  group  began  deliberation  at  the  same  time  WG  3  was  meeting,  a 
generic  set  of  decisions  (that  could  be  supported)  by  analysis  were  used  to 
stimulate  discussion.  Over  the  course  of  the  workshop,  the  group  considered 
WG  3  inputs  as  they  arrived. 

One  notable  possibility  that  was  added  by  the  C4I  domain  was  the  existence  of 
decision  aids  that  are  either  currently  embedded,  or  planned  for  in  near  real-time 
command  and  control  systems.  These  systems  would  benefit  from  analytically 
sound  underlying  algorithms  and  the  group  felt  strongly  that  the  analysis 
community  will  need  to  be  an  active  part  of  their  development. 
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Assertions  (Cont.) 


♦  Accomplishing  these  analytic  missions  require: 

-  Descriptive  tools  (models  and  simulations) 

-  Prescriptive  tools  (choice  and  resource  allocation) 

-  Reliable  and  consistent  data 

♦  A  significant  body  of  work  is  underway  in  the  DoD  that 
advances  the  state-of-the-art  in  modeling  “infrastructure.” 

-  Corrects  deficiencies  inside  mission  areas 

-  Fails  to  capitalize  on  related  work  (schedule  and  resource 
constraints) 

-  This  results  in  inefficient  and  inconsistent  modeling  and  analysis 
efforts. 


The  taxonomy  of  models  we  see  here  include  descriptive  representations  of  the 
military  world  and  also  prescriptive  models  —  those  that  prescribe  a  solution  to 
the  budget  and  capability  trade  space.  We  need  efforts  in  both  arenas  to  produce 
the  analysis  capability  to  serve  the  profession  in  the  future. 

It  was  further  asserted  during  the  opening  session  that  accomplishing  these 
missions  must  be  accomplished  within  shorter  and  shorter  timelines. 
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Analysis  Process  in  Context  of  Knowledge 


ft 


During  the  conference,  we  saw  this  knowledge  management  framework  (the 
blue  boxes)  as  a  hierarchy  of  knowledge  products  built  on  the  base  of  known 
facts  and  derived  meanings.  One  of  the  insights  from  our  work  this  week  is  that 
the  application  of  knowledge  management  within  the  analysis  community  is 
really  a  process  of  becoming  conscious  of  the  importance  of  building  and 
sharing  the  knowledge  of  our  analysis  enterprise  —  both  subject  area  and 
analysis  expertise. 

Knowledge  products  are  produced  by  the  analysis  process  (shown  in  yellow). 
The  essence  of  analysis  has  always  been  to  create  knowledge  from  the  raw 
materials  of  facts  and  meanings. 

One  aspect  of  wisdom  —  the  highest  level  knowledge  product  shown  in  the 
original  framework  (and  missing  from  the  far  right  side  of  this  diagram)  — 
results  from  continuing  to  develop  and  use  analysis  capabilities  over  time. 
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This  is  a  slightly  different  view  of  the  analysis  process  and  products  from  the 
previous  slide.  It  highlights  the  product  areas  that  the  tools  working  group 
might  consider  as  areas  for  improvements. 

Much  of  our  discussion  centered  on  tools  and  data,  particularly  the  need  to 
integrate  information  generated  from  multiple  tools,  sequence  data  flows  from 
one  tool  to  another,  and  to  ensure  that  the  data  objects  were  consistent  such  that 
they  could  be  used  as  is  or  logically  transformed  to  meet  modeling 
requirements.  The  term  “model,”  defined  in  the  broadest  sense  as  the  abstract 
representation  of  any  problem,  was  frequently  discussed  as  a  key  tool. 

Another  emphasis  area  was  to  reduce  the  data-to-tool-to-information  cycle  time 
sufficiently  to  meet  the  needs  of  the  customers  with  very  short  decision 
timelines.  Traditionally,  these  customers  have  relied  on  back-of-the  envelope 
estimates  —  aka  “quick  and  dirty”  analyses  —  to  offer  insights  to  support  their 
decisions.  Since  these  decisions  are  frequently  high-level  and  wide  reaching, 
the  analysis  community  would  like  to  design  tool  and  data  solutions  that  offer 
both  a  high  quality  and  timely  response. 

Decision  support  needs  centered  on  visualization  tools  —  the  ability  to  clearly 
represent  complex  decision  spaces,  dissimilar  tradeoffs,  and  the  like  —  that 
have  not  yet  been  identified.  The  group  identified  KM  practice  as  their 
preferred  solution  option  to  identify  and  reuse  methodologies  and  measures. 
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What  Should  We  Know? 


♦  Assets  on  hand 

-  Authorization/on  hand  data 

♦  Investment  strategy 

-  Equipment  aging. . .  succession  analysis 

-  Personnel  development 

-  Life  cycle  cost  (estimates  and  historical  data) 

♦  OPTEMPO 

-  Current  missions 

-  MTW  readiness/training 

-  Spikes 

♦  Beliefs  about  the  future 

-  Future  OPTEMPO 

-  Personnel  turnover 

-  System  effectiveness 

ms 


One  of  Vince  Roske’s  main  points  in  his  introductory  remarks  was  that  we 
“don’t  know  what  we  know.”  OR  has  in  recent  years  concentrated  on  the 
mechanical  and  physical  aspects  of  predictive  modeling.  This  has  taken  us 
away  from  the  original  roots  of  the  science,  which  was  grounded  in  the 
observation  of  combat  processes  and  their  improvement  through  study. 

We  need  to  return  to  these  roots  and  use  KM  to  acquire  the  information  we  do 
know,  and  catalog  it.  Exercises,  OPTEMPO,  and  force  structure  data  must  be 
treated  as  important  information  that  is  archived  and  made  accessible  to  all 
analysts. 
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Key  is  to  balance  standardization  with  ability 
to  exploit  domain-generated  expertise 

Analysts  are  called  to  serve  a  variety  of  customer,  service,  mission  and 
supporting  function  domains.  This  slide  shows  a  representative  set  of  domains 
—  there  are  others.  The  point  is  that  each  domain  requires  specific  analysis 
perspectives  and  domain  expertise.  Analysis  capabilities  are  developed  within 
these  domains  to  meet  customer  needs.  KM  offers  a  methodology  and  mindset 
to  share  information  across  domains  that  is  represented  here  as  the  center 
“Metadata  Integration  Architecture.” 

“Metadata”  are  data  that  explain  and  establish  the  context  for  data  within  a 
conceptual  system.  One  foundation  of  the  analysis  process  is  to  generalize  or 
abstract  a  problem  to  a  level  where  it  can  be  addressed  with  available  time  and 
resources  and  at  a  level  that  is  consistent  with  a  customer’s  information  needs. 
The  elements  of  these  abstractions  can  be  considered  metadata  and  have  the 
potential  to  be  reused  as  is  or  provide  a  good  starting  point  for  future 
assessments. 

The  point  is  to  exploit  our  body  of  knowledge  to  increase  the  responsiveness 
and  quality  of  analyses  by  making  best  practices  easily  accessible.  The  term 
“standardization”  is  used  as  a  shorthand  for  “data  objects,  algorithms, 
methodologies,  measures  and  other  analysis  elements  that  are  acknowledged 
best  practices.”  The  balance  is  a  delicate  one  —  need  for  communication  across 
domains  with  need  for  flexibility  to  respond  to  customer  needs. 
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The  Multi-Disciplinary  Team  of  the  Future 


Operators 
Decision  Makers 
Planners 

Personnel  Specialists 
Communicators 
Intelligence  Experts 
Logisticians 


Military  OR 
Optimization 
Experimental  Design 
Decision  Analysis 
etc. 


Software  Engineers 
Scientists 

Computer  Scientists 
Comm’s  Engineers 
etc. 


Early  OR  practice  flourished  because  analysts  were  able  to  assemble  multi¬ 
disciplinary  teams,  and  to  bring  structure  to  the  problem  solving  process.  Over 
time  OR  evolved  into  a  discipline  of  its  own,  and  reached  out  to  other 
disciplines  as  needs  were  perceived. 

The  challenges  of  the  21st  century  will  require  advanced,  very  complex 
technologies  in  all  domains.  This  implies  that  no  one  person,  or  discipline,  will 
possess  the  expertise  to  assemble  a  robust  system,  or  even  adequately  describe  a 
complex  problem. 

An  example  of  successful  collaboration  of  this  type  is  taking  place  in  the 
JWARS  development  project.  Descriptions  of  the  JWARS  software 
development  process  and  project  are  available  at  http://www.dtic.mil/jwars. 
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The  group  developed  an  “architecture”  of  systems,  software  and  models  that 
was  perceived  to  be  necessary  to  meet  the  needs  of  military  leaders,  and  defense 
managers  in  the  twenty  first  century.  The  kinds  of  unprecedented  connectivity 
and  consistency  that  are  envisioned  are  required  to  keep  analysis  and  analysis 
products  relevant  inside  the  short  decision  cycles  that  are  emerging.  It  is  also 
necessary  to  achieve  a  culture  of  “knowledge  management”  in  which  the  system 
of  systems  we  are  attempting  to  analyze  has  achieved  a  complexity  that  is 
beyond  the  comprehension  of  any  one  person  or  discipline. 

There  are  three  primary  nodes  in  this  network  of  systems.  First,  there  are  near 
real-time  command  and  control  systems.  These  systems  are  used  by  operational 
planners  and  commanders  that  are  tasked  to  execute  the  nation’s  military 
missions.  In  the  information  age  military,  they  exist  in  some  form  or  another 
from  the  NMCC,  down  to  the  soldier  in  the  foxhole.  The  second  node  is 
analysis  models.  These  models,  ranging  from  simple  spreadsheets  and  decision 
tables  to  large  theater  simulations  are  used  to  develop  and  evaluate  plans, 
choose  courses  of  action,  and  manage  the  DoD.  The  final  node  envisioned 
consists  of  training  simulations.  Again,  these  simulations  vary  in  scope  and 
complexity. 

All  of  these  nodes  have  a  common  requirement.  They  all  need  data  (both 
primitive  and  meta)  to  function.  If  we  are  to  achieve  the  envisioned 
interoperability  it  is  absolutely  essential  that  this  data  be  logically  consistent  and 
portable  across  domains  when  desired. 
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One  fact  that  emerged  is  that  the  C4I  community  is  embedding  decision  aids 
into  emerging  command  and  control  systems.  These  decision  aids  are 
effectively  models  that  operational  staff  members  and  commanders  will  use  to 
make  decisions  on  those  short  timelines  we  see  emerging.  It  is  important  that 
analysts  (and  others)  be  involved  in  the  development  of  these  algorithms  to 
increase  the  likelihood  that  the  underlying  logic  is  sound  and  consistent. 
Moreover,  this  kind  of  involvement  is  necessary  if  the  DoD  is  to  achieve  its 
objectives  with  regard  to  the  use  of  software  and  achieve  a  more  cost  effective 
maintenance  stream  for  models  and  data  of  all  types. 

In  theory  the  DII  COE  architecture  provides  vehicles  for  data  communication 
from  live  systems  to  simulations.  However,  it  was  learned  that  the  current  data 
representations  in  the  C4I  systems  and  analytic  models  will  not  support  this 
objective.  Moreover,  there  are  similar  inconsistencies  in  the  data  models  used 
between  the  C4I  and  training  domains. 
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An  example  illuminates  how  this  system  would  improve  our  operational 
decision  making.  Envision  a  requirement  for  CINCPAC  to  deploy  humanitarian 
assistance  to  East  Timor.  There  is  a  need  to  provide  logistical  infrastructure  to 
rebuild  or  secure  this  nation.  Also,  assume  the  planning  process  produced  a 
contingency  plan  that  includes  identification  of  the  forces  that  would  be  sent  on 
this  mission. 

The  plan  and  all  of  its  assumptions  have  been  described  in  some  analysis  model 
(Perhaps  in  an  OPLAN  Format)  and  reside  in  data.  At  the  time  the  requirement 
to  execute  is  identified,  the  data  in  the  plan  are  compared  to  ground  truth  data. 
The  army  database  on  current  forces  identifies  the  fact  that  the  25th  Supply  and 
Transport  Battalion  which  was  in  the  plan  is  not  available  because  it  is 
deployed;  and  the  TRANSCOM  data  shows  that  the  C-5  that  are  necessary  to 
meet  the  timeline  are  committed  to  some  other  high  priority  mission.  What  has 
happened  is  we  have  used  existing  data  to  quickly  illuminate  assumptions  that 
were  valid  at  planning  time,  but  are  not  at  execution  time.  Operators  are  then 
quickly  able  to  focus  on  managing  the  exceptions. 

Added  to  the  previous  vision  are  data  models  for  each  of  the  three  primary 
nodes.  It  is  essential  to  realize  that  there  are  separate  and  distinct  data  models 
that  must  be  aligned  and  synchronized  to  achieve  interoperability. 

The  deliberations  in  the  group  revealed  that  there  are  fundamentally  no 
significant  technological  impediments  to  achieving  this  vision.  The  obstacles 
largely  relate  to  achieving  a  sound  common  data  model,  and  developing  a 
“cultural  will”  to  expose  data  to  other  domains. 
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WG  #4  New  Analysis  Tools  and  Methods 

Emerging  Developments  in  Analytical  Approaches  (1  of  6) 

♦  The  Army  standards  category  coordination  process  appears  to  have 
potential  for  improving  the  consistency  of,  and  access  to,  modeling 
resources  for  analysis  and  reuse. 

-  Air  Force  and  Navy  are  developing  similar  approaches. 

-  The  standards  structure  provides  a  foundation  for  achieving 
knowledge  management. 

-  Encodes  analysis  domain  expertise. 

-  Could  migrate  into  a  best  practices  compendium. 

-  Standards  need  to  be  well  documented  (open  peer  review,  clarity). 

-  Process  is  as  (or  more)  important  than  the  resulting  standard. 


The  Issue  is  Good  Meta-Data 


The  Army  presented  their  evolving  standards  category  coordination  process.  It 
is  ongoing  and  both  the  Navy  and  AP  have  undertaken  efforts  to  develop  similar 
programs.  These  efforts  provide  a  foundation  for  KM  rules  and  provides 
domain  area  expertise.  It  is  an  existing  metamodel  of  analysis  domain 
knowledge.  Moreover,  the  structure  provides  an  emerging  link  to  the  training 
and  C4I  domains.  These  efforts  are  emerging  because  there  is  a  recognized 
need  to  achieve  consistency  across  domains,  if  other  efficiencies  promised  by 
technology  are  to  be  realized. 

The  group  also  provided  “seeds”  to  support  further  thought  on  this  concept,  so 
that  one  could  possibly  avoid  some  of  the  pejorative  connotations  that 
accompany  the  term  “Standards.” 
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WG  #4  New  Analysis  Tools  and  Methods 

Emerging  Developments  in  Analytical  Approaches  (2  of  6) 


♦  Conceptual  modeling  and  MSRR  efforts  appear  to  provide  a  good 
prototype  for  the  content  of  a  DoD  M&S  knowledge  management 
system  with  regards  to  military  force  behavior: 

-  DMSO  MSEA  investments  provide  a  foundation  (direction)  for 
interoperability/consistency. 

»  C4I  compatibility  is  a  significant  shortcoming  (e.g.  terrain). 

»  Effort  is  in  early  stages. 

-  There  is  a  significant  issue  associated  with  opaque  (undocumented) 
data  generation  and  transformation. 

»  “Life  cycle”  costs  of  models 

»  Data  stream 

-  IWEDA  and  NRMM  are  examples  of  effective  reuse  by  C4I  and 
analysis. 

(Hi 


The  conceptual  modeling  and  MSRR  efforts  were  identified  as  a  method  for 
encoding  force  behavior  domain  knowledge.  Group  members  identified  efforts 
around  the  community  that  are  similar  to  the  work  that  was  presented.  The 
group  identified  weaknesses  in  the  technology  but  each  implementation  brings 
the  community  closer  to  describing  force  behavior  in  a  manner  that  will  support 
more  efficient  modeling,  and  consistent  data  exchange. 

Two  examples  of  effective  reuse  across  domains  exist  in  JWARS.  One  is  the 
Engineer  Research  and  Development  Center  Vicksburg’s  application  of  the 
NATO  Reference  Mobility  Model  in  both  C4I  systems,  and  simulation.  They 
map  characteristics  of  trucks  to  trafficability  networks.  Use  of  this  “model”  is 
also  envisioned  in  WARSIM,  the  land  component  of  JSIMS.  ARL’s  Integrated 
Weather  Effects  Decision  Aid  (IWEDA)  is  being  used  in  command  and  control 
as  well  as  JWARS.  This  attempt  at  reuse  identified  shortcomings  that  are  now 
identified  research  objectives  of  the  Army  Research  Lab  (ARL). 
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WG  #4  New  Analysis  Tools  and  Methods 

Emerging  Developments  in  Analytical  Approaches  (3  of  6) 


♦  There  are  technologies  emerging  to  better  “automate”  data 
generation  and  management. 

-  To  wit:  Army  Flow  Model  and  JDS  Data  Warehouse. 

-  We  lack  a  consistent  understanding  of  “source”  data  and  how  to 
propagate  it. 

-  DoD  Standard  Data  Dictionary  is  a  start. 

-  Inconsistency  between  C4I  systems  and  models. 

-  Hope  exists  (ERDC’s  NRMM  is  “consistent”). 


The  JDS  data  warehouse  presentation  shows  technology  that  can  achieve  a 
“central”  repository  for  data  management  and  distribution.  Web  technologies 
could  in  fact  be  used  to  distribute  this  repository  so  that  domain  experts  retained 
control  over  their  data. 

The  Army  has  a  tool  called  the  Flow  Model  that  deconflicts  their  standard 
databases  —  e.g..  lets  us  know  how  many  trucks  are  in  the  army  today.  This  is 
the  kind  of  information  that  should  be  more  broadly  available  to  support  both 
planning  and  execution. 

The  DoD  data  dictionary  that  exists  in  the  C4ISR  world  is  not  consistent  with 
the  objects  typically  found  in  analytical  models  and  simulations. 
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WG  #4  New  Analysis  Tools  and  Methods 

Emerging  Developments  in  Analytical  Approaches  (4  of  6) 

♦  There  are  some  “logically  sound”  prescriptive  approaches  to 
supporting  decision  makers  that  are  emerging  or  have  had  staying 
power. 

-  Marine  POM  decision  analysis. 

-  CAA’s  “Analysis  in  a  Rucksack.” 

-  Features  of  success  are: 

»  Simple,  sound  logic. 

»  The  ability  to  accommodate  multiple  stakeholders. 

»  Transparency. 

»  Visualization.... consistency  and  clarity. 


The  group  talked  a  lot  about  the  need  for  quick  and  dirty  tools  that  support 
decision  makers  and  ability  to  reduce  knowledge  to  context  of  decision. 

Recommendations  vice  data. 

USMC  decision  analytic  approach  is  23  years  old.  The  stuff  exists  that  could  be 
used  or  adapted  for  the  toolkit. 

There  are  properties  of  these  tools  and  analyses  that  cause  them  to  endure. 
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WG  #4  New  Analysis  Tools  and  Methods 

Emerging  Developments  in  Analytical  Approaches  (5  of  6) 


♦  The  application  of  “quick  and  dirty”  and  “presentation”  tools  is 
a  weakness  in  the  analytic  practice. 

♦  Discipline  and  commercial  tools  can  contribute. 

-  Value  focused  thinking. 

»  Fundamental  objectives  (hierarchy). 

»  Means  objectives  (network). 

-  Commercial  “modeling”  tools. 

»  Logical  decisions. 

»  Analytical....  Etc. 

♦  Data  issues  are  a  recurring  theme. 


The  manner  in  which  quick  and  dirty  tools  are  applied  is  a  weakness  in  the 
current  military  OR  profession.  The  historical  reliance  on  large  scale  combat 
simulations  as  the  primary  “validated”  approach  to  conducting  analysis  lacks  the 
flexibility  to  address  the  broad  range  of  challenges  the  DoD  currently  faces. 

That  said,  there  are  examples  of  successful  analytic  projects  that  do  not  rely  on 
the  large-scale  simulations  for  achieving  insight.  A  common  theme  of  these 
successful  efforts  is  they  are  focused  on  the  decision  situations  most  relevant  to 
the  decision  makers,  they  are  transparent,  and  they  are  based  on  sound, 
consistent  data. 
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WG  #4  New  Analysis  Tools  and  Methods 

Emerging  Developments  in  Analytical  Approaches  (6  of  6) 


♦  There  are  advanced  modeling  concepts  that  appear  to  have 
some  potential  for  solving  some  of  our  “hard”  modeling 
problems. 

-  Agent  based  simulation. 

»  Human  decision  making. 

»  Soft  factors. 

-  Mathematical  approaches  to  fusion  (sensor  and  data). 


m 


There  are  some  hard  problems  out  there  that  the  group  agreed  would  bear  large 
fruit  if  we  could  tackle  them  and  apply  some  quantitative  science  to  them. 
Human  decision  making  is  not  well  represented  today  in  analytical  processes 
and  models.  Finally,  there  is  some  highly  important  Bayesian  fusion  work  that 
holds  the  potential  to  develop  sensor  and  multi-resolution  theory.  This  work  will 
also  advance  the  ability  of  analysts  to  study  and  assess  ISR  activities. 
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WG#4  New  Analysis  Tools  and  Methods 

Potential  Applications  for  Each  New  Approach 


♦  The  identified  approaches  provide  a  foundation  that  can  support 
multi-disciplinary  teams  in  meeting  new  ANALYSIS 
challenges. 

-  “Warm”  consistent  data  set. 

-  “Modular”  set  of  models  to  draw  on  and  build  from. 

♦  There  are  holes  (there  always  will  be). 

-  New  approaches  can  be  added  to  the  base  of  knowledge. 

-  Commercial  tools  can  provide  a  “workbench.” 


Applications:  ran  some  vignettes.  Seemed  to  be  able  to  map  to  all  of  them. 
Want  over  50%  solution  on  data,  80%  solution  on  models. 

Will  always  be  new  analytic  challenges.  Need  to  build  foundation. 
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WG  #4  New  Analysis  Tools  and  Methods 

Near  Term  Recommendations  (1  of  3) 

♦  Increase  DoD  emphasis  on  supporting  inter-service  collaboration 
toward  standards. 

-  This  builds  on  the  existing  service  efforts. 

-  Possible  organizational  models  are: 

»  Existing  JTCG  (JMEM’s)  is  a  possible  model. 

»  MSRR  board  of  directors. 

-  This  should  include  Joint  rules  for  reconciliation  of  standards. 

-  Establish  a  baseline  (i.e.  a  minimal  standard  for  meta-data). 

♦  Use  limited  experience  as  a  foundation  for  KM  implementation. 

-  Fonnal  plan  for  achieving  the  vision. 

»  Senior  leader  support  and  involvement. 

»  Deliberate  investment  strategy. 

»  Uses  the  multi-disciplinary  team. 

»  There  should  be  a  co-evolution  of  culture  and  funding  for 
technology. 


Work  is  starting;  let’s  get  some  resources  into  it.  Needs  joint  rules  to  reconcile 
standards  —  not  saying  use  same  set  but  must  be  logically  consistent. 

Need  to  build  a  formal  plan  not  just  throw  money  at  the  problem  and  touch  all 
the  communities. 

As  plans  become  more  complete  and  culture  evolves,  fund  accordingly 
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WG  #4  New  Analysis  Tools  and  Methods 

Near  Term  Recommendations  (2  of  3) 


♦  The  Analysis  community  should  be  represented  in  the  DII  COE 
process. 

♦  The  DoD  should  develop  and  resource  a  strategy  for  collecting 
“other  source”  data,  and  applying  KM. 

-  NTC  exercise  data. 

-  Historical  deployment  data. 

-  Lessons  learned  data. 

-  etc. 


(m 


Decision  aids  being  developed  in  C2.  We’re  either  there  or  miss  the 
opportunity. 

Lessons  learned  hits  come  from  behind  the  iron  curtain. 
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WG  #4  New  Analysis  Tools  and  Methods 

Near  Term  Recommendations  (3  of  3) 

♦  Encourage  the  creation  and  use  of  “quick  and  dirty”  tools.  This 

should  be  accompanied  by: 

-  Rules  of  the  road  for  using  data. 

-  Rest  practices. 

-  Mechanisms  for: 

»  Timely  peer  review. 

»  Documenting  use. 

"T  Knowledge 

»  Allowing  evolution. 

Management  ^ 

x.  Culture 

The  working  group  shared  a  common  vision  that  the  community  should  let  “a 
thousand  flowers  bloom”  with  respect  to  quick  and  dirty  tools  that  lie  outside 
the  formal  modeling  constructs.  These  tools  hold  the  potential  to  free  us  from 
the  constrained  environment  of  the  large-scale  simulation  and  return  some  speed 
and  flexibility  to  our  science.  The  problem  with  this  philosophy,  however,  is  to 
ensure  accuracy  and  consistency  in  results  while  allowing  flexibility  in 
application. 

Rules  of  the  road  for  using  data  would  allow  the  sources  of  data  to  have  a  say  in 
the  use  of  their  numbers,  while  a  “Best  Practices”  guide  would  help  to  maintain 
some  consistency  in  approach.  Finally,  the  real  value  of  the  KM  paradigm 
comes  clear  in  providing  a  mechanism  for  peer  review  and  documenting  the 
usage  of  technique.  This  should  speed  re-use  and  allow  for  rapid  evolution  of 
technique. 
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WG  #4  New  Analysis  Tools  and  Methods 

Recommendations  for  Future  Research 

♦  A  detailed  review  should  be  conducted  that  maps  ongoing  research 
to  identified  shortcomings: 

-  Advanced  concepts  that  “automate”  human  decision  making 
(C2). 

-  Intelligent  Agents. 

-  Inference/Fusion. 

-  Visualization. 

-  etc. 

♦  An  investment  strategy  should  then  be  developed  to  fill  the 
identified  holes. 


The  DoD  must  invest  in  the  research  to  allow  us  to  quantify  the  human  decision 
making  process,  use  intelligent  agents,  and  tackle  the  fusion  problem  so  as  to 
better  represent  intelligence  processes  in  analysis.  Visualization  techniques, 
both  to  increase  human  absorption  of  information  and  to  yield  better  displayed 
information  for  decision  makers,  must  also  be  investigated  with  energy. 

The  Department  must  conduct  a  comprehensive  review  that  maps  research  into 
identified  shortfalls.  We  don’t  know  what  we  don’t  know,  about  the  basic 
research  being  conducted  in  these  areas.  Before  we  create  an  investment 
strategy,  we  need  to  get  the  holes  right  and  figure  out  how  to  fill  them. 
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The  future  framework  will  require  a  warm  bed  of  consistent  data  and  models 
that  will  operate  inside  the  existing  decision  cycles.  The  vision  is  to  achieve  a 
fairly  seamless  linkage  between  real  systems  and  the  families  of  training  and 
analytical  models  and  simulations. 

Each  component  has  its  own  data  (represented  in  purple)  that  are  particular  to 
its  domain  and  independent  of  the  other  areas.  This  data  is  highly  relevant  and 
important,  and  must  be  available  to  all  through  the  KM  paradigm. 

Analysis  methods  are  important,  but  the  key  to  turning  around  analysis  more 
quickly  is  data.  A  warm  bed  of  data,  COTS  tools,  plus  a  robust  training  program 
will  help  us  utilize  what  we’ve  got.  Imbedded  tools  in  C4I  systems  will 
generate  displays  —  need  to  help  design  these. 

Changing  the  tools  will  not  speed  the  analysis  process  alone  —  we  need  to 
change  the  foundation  that  we’re  working  with  to  give  more  speed.  KM  is  key 
here.  CINCS  have  existing  contingency  plans,  and  analysts  often  find  the 
closest,  most  relevant  plan  to  build  from.  We  need  a  richer  set  of  things  to  work 
with,  and  netted  real  world  and  analytical  systems  in  a  KM  paradigm  will  give 
us  this. 

Over  time,  models  will  generate  information  that  becomes  another  data  source. 
Models  would  generate  new  metadata  that  should  be  included  in  this  picture. 
Analysis  cases  we  build  can  live  on  the  shelf  ready  for  reuse. 
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MORS  Workshop 


Evolving  the  Practice  of  Military 
Operations  Analysis  in  DoD 

Synthesis  Group 

Chair:  Dr.  Stuart  Starr 

Co-Chairs:  Dr.  Jerry  Kotchka 
and  Dr.  Tom  Allen 


February  29,  2000  through  March  2,  2000 
Naval  Postgraduate  School,  Monterey,  CA 


This  presentation  provides  the  comments  and  insights  of  the  Synthesis  Group 
for  the  “Evolving  the  Practice  of  Military  Operations  Analysis  in  DoD” 
workshop  as  presented  during  the  final  session.  The  Synthesis  Group  attempted 
to  pull  together  common  themes  from  all  the  working  groups,  as  well  as  to 
provide  insights  about  how  separate  themes  developed  by  these  groups  might  be 
harmonized  to  support  a  common  vision  and  meet  the  overall  objectives  of  the 
workshop. 
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Synthesis  Team 


♦  Dr.  Stu  Starr,  FS  (Chair  Synthesis  Emeritus)  -  MITRE 

♦  Dr.  Jerry  Kotchka  -  Co-chair  -  LM/NESS 

♦  Dr.  Bob  Hinkle  (WG-1)  -  DUSA(OR) 

♦  Dr.  Tom  Allen  (WG-2)  -  IDA 

♦  Dr.  Mike  Sovereign  (WG-3)  -  IJWA/NPS 

♦  Dr.  Mark  Youngren  (WG-4)  -  MITRE 

♦  Dr.  Roy  Rice  (Executive  Council  Rep)  -  TBE 


Members  of  the  team  are  listed  on  this  slide.  Dr  Stu  Starr,  FS,  Working  Group 
Chair,  was  called  away  the  day  before  the  workshop  opened  and  was  unable  to 
attend  in  person.  Despite  that,  his  massive  research  and  coordination  with  the 
other  working  group  chairs  leading  up  to  the  workshop  enabled  him  to  gather  a 
great  deal  of  background  material,  as  well  as  to  provide  an  initial  shape  to  this 
outbrief.  Without  his  participation  and  virtual  leadership,  the  final  product  of 
the  group  would  have  suffered  immensely.  Dr  Jerry  Kotchka  stepped  in  to  ably 
coordinate  the  activity  in  Monterey.  The  team  included  members  from  each 
working  group  and  met  daily  with  the  chairs  and  cochairs  of  those  groups,  both 
to  highlight  common  areas  and  to  request  effort  on  specific  topics  that  would 
help  the  Synthesis  team  in  their  efforts  to  create  a  follow-on  action  plan  to  help 
the  military  operations  research  community  achieve  the  overall  vision  of  the 
conference. 
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Toward  this  end,  the  Synthesis  Group  asked  each  Working  Group  (WG)  to 
articulate  a  vision  that  would  capture  the  end  state  resulting  from  full 
implementation  of  their  group’s  recommendations.  Based  on  their  inputs,  the 
Synthesis  Group  articulated  the  vision  shown  here.  We  believe  this  vision 
transcends  and  incorporates  the  WG  visions  and  describes  how  emerging 
capabilities  such  as  collaboration  and  knowledge  management  can  transform  the 
practice  of  military  analysis.  The  emergence  of  powerful  communication  and 
computer-based  capabilities  is  already  changing  the  economic  world.  Where 
before  the  focus  was  on  production,  the  emerging  giants  of  e-commerce  are  now 
succeeding  through  their  acquisition  and  management  of  information.  The  most 
successful  and  most  profitable  of  these  businesses  are  those  that  enable  the 
highest  level  of  information  sharing,  so  that  opportunities  are  quickly  identified, 
decisions  made  based  on  the  full  amount  of  information  and  insight  available  in 
the  work  force,  and  then  action  initiated  appropriately.  The  knowledge  resident 
in  every  worker  is  rapidly  connected  to  the  decision  makers  so  that  timelines  are 
reduced  and  customer  needs  satisfied,  optimizing  company  effort  and  taking  full 
advantage  of  the  knowledge  base  of  the  organization.  In  a  like  manner,  military 
analysis  in  the  future  will  need  to  be  fully  integrated,  with  analysts  and  decision 
makers  so  closely  linked  through  technical  and  personal  means  that  the 
maximum  benefit  will  flow  to  the  organization.  In  fact,  the  interaction  will  be 
such  that  the  final  results  of  any  study  will  surprise  no  one,  since  decision 
makers  and  analysts  will  both  be  engaged  throughout  the  analytic  process, 
allowing  the  strengths  and  insights  of  each  to  be  fully  incorporated.  Such  a 
construct  will  revitalize  the  practice  of  analysis,  take  advantage  of  the  emerging 
capabilities,  and  better  address  new  challenges  confronting  analysis  in  the  new 


millennium. 
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Analysis  to  Support  Decision  Making 


Decision  Making  —  choosing  among  alternatives 
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This  slide  illustrates  one  view  of  the  current  practice  of  analysis.  It  shows  that 
analysis  is  focused  on  one  goal:  helping  decision  makers  choose  among 
alternatives.  The  four  elements  analysts  support  in  this  activity  include:  1) 
helping  the  decision  maker  define  the  issue  or  questions;  2)  helping  to  define  the 
decision  space  so  that  a  full  range  of  alternatives  can  be  considered;  3) 
establishing  meaningful  measures  of  effectiveness  and  criteria  that  will  help 
show  the  strengths  and  weaknesses  of  the  various  alternatives;  and,  4)  then 
creating  understandable,  traceable  and  repeatable  methodologies  that  will  assist 
in  this  selection  process.  To  do  this,  an  analytic  foundation  must  be  in  place. 
This  foundation  includes  the  people  who  contribute  to  the  process,  primarily 
analysts  (but  including  a  broad  range  of  other  scientists,  engineers  and  subject 
matter  experts);  models,  or  the  concepts,  mathematical  abstractions  or  other 
constructs  that  are  used  to  decompose  a  problem  and  focus  on  key  aspects;  and 
the  data,  scenarios,  assumptions  and  other  factors  that  are  necessary  to  support 
the  process.  The  issue  addressed  by  this  workshop  is  how  this  framework  is  and 
will  be  impacted  by  the  changes  in  the  National  Security  and  Technological 
Landscapes  and  how  the  practice  of  analysis  should  evolve  to  handle  the 
challenges  and  take  advantage  of  the  opportunities  provided  by  these  changes. 
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The  Analyst’s  Dilemma 


“Whatever  happened  to  elegant  solutions?” 

BS 


Of  course,  the  very  complexities  presented  by  the  rapidly  changing  landscapes 
associated  with  this  environment  may  have  removed  the  ability  for  analysts  and 
others  to  focus  on  the  big  picture.  The  new  environment  as  assessed  by 
Working  Group  3  presents  a  conundrum:  we  are  asked  to  provide  insight  on  the 
complex  interrelationships  between  factors  input  to  and  within  our  models  (that 
reflect  interrelationships  in  the  real  world),  but  we  are  also  asked  to  reduce  the 
complexity  and  increase  the  transparency  of  our  tools  and  analysis.  Although  it 
may  not  be  possible  or  necessary  to  reduce  the  complexity  of  our  high-end 
models  (recognizing  of  course  that  not  all  analysis  requires  large  and  complex 
models),  open  sharing  of  model  data,  assumptions,  algorithms,  etc.,  all  within  a 
set  of  agreed-upon  standards  and  implemented  through  knowledge  management, 
may  increase  the  transparency  of  the  models  and  analysis  process  among  trained 
analysts.  The  analyst  will  always  have  to  convert  complex  mathematical  and 
structured  logic  into  forms  understandable  to  a  decision  maker  without  an  OR 
background  in  all  but  the  simplest  cases.  Thus,  an  elegant  solution  may  not  be  a 
reduction  in  complexity  to  a  handful  of  variables,  but  instead  be  a  transparency 
and  sharing  among  analysts  that  is  communicated  to  the  decision  maker 
throughout  the  analysis  (rather  than  at  the  end)  in  a  clean  and  direct  (if  not 
elegant)  manner. 
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Definition 


Dr.  Ruth  David’s  definition  of  “Knowledge 
Management” 

-  Process  through  which  an  enterprise  uses  its  collective 
intelligence  to  achieve  strategic  objectives: 

»  Strategy 
»  People 
»  Process 
»  Technology 


S3 


At  the  same  time,  there  are  potential  solutions  available,  at  least  to  address  part 
of  the  changing  environment,  and  to  help  our  analytic  community  and  other 
parts  of  DoD  enterprise  to  adapt  and  prevail  in  the  emerging  environment  of  the 
21st  century.  Dr.  Ruth  David,  our  keynote  speaker,  defined  knowledge 
management  as  shown  on  this  slide.  Inherent  in  her  definition  was  the 
understanding  that  the  enterprise  using  knowledge  management  was  a  “lean, 
mean,  fighting  machine  team”  focused  on  common  objectives  and  with  a  shared 
understanding  of  the  necessity  and  value  of  each  other’s  contribution.  Such  a 
definition  of  enterprise  may  not  reflect  the  current  state  of  DoD,  where  often 
mutual  contributions  are  not  recognized  and  parochialism  can  break  down 
teamwork.  However,  to  the  extent  that  shared  goals  and  team  work  reflect  DoD, 
and  in  the  areas  that  knowledge  management  can  help  the  department  move 
toward  this  enterprise  vision,  knowledge  management  and  its  implementation 
surely  should  be  embraced  by  the  department. 
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The  accompanying  figure  provides  a  business  process  re-engineering 
perspective  of  collaboration  and  knowledge  management.  The  Synthesis  Group 
concluded  that  if  we  are  to  successfully  implement  these  capabilities  in  a 
dynamic  environment,  we  must  consistently  address  all  of  these  factors.  For 
example,  attempting  to  field  a  technology  that  makes  available  all  information 
to  all  members  in  a  culture  where  classification,  close  hold  consideration  of 
alternatives,  and  fear  that  knowledge  of  a  programmatic  weakness  could  result 
in  elimination  of  a  system  even  if  other  elements  are  robust  and  essential  to  a 
military  capability,  can  only  result  in  failure.  Before  the  technology  can  be 
designed,  the  culture  must  be  addressed  and  pragmatic  steps  taken  to  show  how 
such  information  sharing  can  actually  result  in  overall  benefits  to  the 
participating  organizations  and  people.  Concrete  trust  building  steps  can  be 
followed  by  bigger  steps  ultimately  implementing  the  full  capability.  Just  as 
culture  must  be  addressed,  so  must  the  inherent  goals  and  objectives  of  the 
involved  organizations  and  people.  Many  experts  at  this  workshop  agreed  that 
the  technological  solutions  for  most  of  the  knowledge  management  and 
collaboration  techniques  are  already  in  hand.  The  biggest  challenges  will  be  in 
addressing  the  needs  of  culture,  organization  and  people. 
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Emerging  Pressures 


Analyst  (analysts,  scientists,  engineers,  etc) 
Models  (abstraction  of  reality) 


Returning  to  our  analytic  construct,  we  can  see  that  changes  in  the  national 
security  landscape  are  already  shaking  some  of  the  analytic  pillars.  Evolving 
from  attrition-based  warfare  to  effects-based  warfare,  understanding  the 
implications  of  C4ISR  systems  and  capabilities,  and  the  emergence  of  a  broad 
new  array  of  asymmetric  threats  are  all  impacting  the  ability  of  the  analytic 
community  to  develop  reasonable  measures  of  effectiveness  as  well  as  to 
develop  clear  and  repeatable  methodologies  to  assess  these  aspects  of  military 
activity.  While  these  areas  are  outlined  with  the  dashed  lines  (which  appear  in 
red  in  color  versions  of  the  chart),  the  technological  landscape  and  emergence  of 
new  capabilities  like  KM  are  impacting  the  rest  of  the  words  to  potentially 
transform  the  way  we  conduct  analysis. 
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Challenges 

%V  National  Security  Landscape*** 

-  Vfli^sTng" effects  (C4ISR  follow-on  meeting) 

-  Environmental  -  Threat  (world  is  changing  -  are  we?) 

-  Ash  Carter  and  Bill  Perry  (A/B/C  Lists  from  QDR  Workshop) 

-  Technology  insertion 

-  QDR  2001  (strategy  -  ???) 

♦  Technology  landscape 

-  Knowledge  management 

-  Collaborative  analysis  management 

-  Commercial  initiatives 


This  chart  provides  more  detail  on  the  changes,  with  the  circled  “National 
Security  Landscape”  reflecting  the  circled  area  on  the  last  chart.  We’ve  already 
hinted  at  the  changing  paradigm  of  massed  armies  meeting  on  a  battlefield  to 
the  massing  of  effects  at  the  appropriate  time  and  place  as  the  way  current  US 
military  leadership  is  starting  to  think  about  warfare.  In  fact,  to  help  think 
through  the  implications  of  such  changes,  MORS  has  conducted  a  number  of 
workshops  and  will  hold  a  C4ISR  Follow-on  Workshop  in  the  fall  to  continue  to 
evolve  our  understanding  in  this  arena.  Some  of  our  speakers  in  this  workshop 
have  referred  to  the  Ash  Carter/Bill  Perry  work  on  who  the  threats  of  the  future 
are.  They  range  from  an  A  List  of  direct  threats  to  national  survival  which 
would  include  homeland  attack  by  catastrophic  terror  agents,  to  a  B  List  of 
threats  to  national  interests  that  the  current  strategy  seems  to  address,  to  a  C  List 
of  important  problems  that  do  not  threaten  US  vital  interests  (e.g.  Kosovo, 
Somalia).  In  addition,  new  technologies  require  new  analytic  capabilities  to 
measure  the  effects  and  potential  contributions  from  such  areas  as  directed 
energy,  information  technologies,  unmanned  systems  and  space.  The  QDR 
process  may  change  the  national  military  strategy,  so  the  analytic  profession 
will  need  to  be  responsive  both  in  helping  to  determine  the  implications  of 
strategy  shifts  as  well  as  in  implementing  any  new  strategy  in  the  analyses  of  the 
future. 

Just  as  the  security  landscape  is  changing,  so  is  the  technology  environment  in 
which  analysis  is  conducted.  This  workshop  has  highlighted  KM  and 
collaborative  capabilities,  but  there  are  a  number  of  other  initiatives,  such  as 
simulation  based  acquisition  and  other  e-commerce  activities  that  the  analytic 
community  needs  to  track  and  adopt  where  it  makes  sense  to  do  so. 
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This  slide  shows  an  illustration  of  an  ideal  end-state  for  the  current  efforts  to 
improve  analytical  practices.  There  are  three  families  of  models  and 
simulations:  M&S  embedded  in  real-time  command  and  control  systems; 
training  and  exercise  support  simulations;  and  models  and  simulations  used  to 
support  analysis  (to  include  T&E  analysis  and  analysis  of  non- warfighting 
issues  such  as  infrastructure  and  business  process  improvement).  Within  each 
domain  are  a  set  of  models  and  simulations  tailored  to  the  purpose  at  hand.  No 
single  model  will  apply  universally  across  all  domains,  but  in  this  ideal  end  state 
we  have  data,  software  objects,  and  process  representations  shared  across  the 
domains  where  it  is  logically  consistent  with  the  model  and  purpose. 

Infrastructure  programs  currently  under  development  are  assumed  to  be  in  place 
in  this  end  state,  to  include  the  High  Level  Architecture  (HLA)  for  simulations 
and  the  Defense  Information  Infrastructure  Common  Operating  Environment 
(DII  COE)  linking  all  areas  together.  HLA  federations  which  may  include 
simulations  widely  distributed  over  space  will  exist  where  appropriate,  but  these 
federations  should  be  standardized  and  validated  for  consistency  across 
federates  before  use.  The  creation  of  “on  the  fly”  federations  that  grab  objects 
and  federates  scattered  across  the  DII  to  create  unique  environments  is  probably 
still  not  possible  in  this  end  state  because  there  is  no  guarantee  that  the 
representations  will  be  internally  consistent  (consistency  is  not  assured  even  if 
data  is  passed  through  an  HLA),  and  it  is  unlikely  that  a  single  analyst  or  team 
located  at  a  single  site  will  be  familiar  with  all  of  the  objects  or  federates 
potentially  available.  Instead,  these  simulations  may  be  combined  through  the 
use  of  collaborative  analysis  rather  than  software  linkages. 
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All  of  the  tools  are  presumed  to  be  in  a  bed  of  warm  data.  That  is,  there  is  a  data 
infrastructure  that  makes  data  readily  available  (with  exhaustive  metadata  explaining  the 
derivation  and  accredited  use  of  the  data)  for  all  of  these  systems.  For  example,  an 
analysis  of  real  world  capability  should  be  able  to  draw  on  the  same  data  set  showing 
current  status  and  deployments  of  units  that  a  real-time  command  and  control  system 
might  use.  Similarly,  an  analysis  of  a  future  capability  (for  example,  a  2010  force  in  a 
particular  scenario)  should  be  able  to  draw  on  a  validated  and  commonly  used  set  of  data 
to  represent  the  scenario,  system  performance,  OPFOR  performance,  etc.  Generation  and 
storage  of  this  data  requires  a  significant  resource  commitment  within  the  Department  of 
Defense,  but  it  is  necessary  to  achieve  the  desired  end  state. 

A  key  feature  of  this  environment  is  that  the  output  of  the  models,  as  well  as  the  results  of 
specific  analyses  and  the  lessons  learned  from  training  and  exercises,  are  also  added  to  the 
common  data  base.  This  provides  a  useful  source  of  information  that  can  be  of  particular 
help  to  quick  tum-around  analyses,  which  may  synthesize  the  outputs  of  previous  models, 
exercises  and  analyses. 


Synthesis  WG  - 11 


Tools  and  Techniques/Challenge  Matrix 
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New  Wars  =  Methods  for  analyzing  new  domains  of  Warfare 

;Behavioral=  Inclusion  ofbehavioral  factors  in  current  models 

jSupport=Masure  combat  impact  of  support  (Log,  C4ISR,  OPTEMEPO,  Readiness  etc) 

^Interactions  « Identifying  comples  interrelationships  between  factors 

iRis k=  Quantification  ofriskand  uncertainty 

INon  Mode  1=  Reduce  the  model  centricity  of  analysis 

Part  of  the  improved  environmental  end  state  is  the  development  of  new  tools, 
techniques  and  methodologies  to  address  the  specific  challenges  of  the  future. 
Some  of  the  new  or  emerging  tools  and  techniques  identified  by  Working  Group 
(WG)  4  are  shown  on  this  chart  to  illustrate  which  show  promise  to  address,  at 
least  in  part,  the  challenges  identified  by  Working  Group  3.  Those  challenges 
fell  into  6  broad  categories:  1)  changing  characteristics  of  warfare  in  the  21st 
Century  (New  Wars);  2)  inclusion  of  the  effects  of  humans  and  human  factors 
(e.g.,  morale,  training)  into  models  (Behavioral);  3)  showing  the  effect  of 
logistics  and  other  support  activities  on  warfighting  outcomes  (Support);  4) 
identification  of  complex  interrelationships  between  input  variables  and 
between  internal  variables  (processes)  within  models  (Interactions);  5)  explicit 
quantification  of  risk  and  uncertainty  in  analysis  and  in  communications  with 
decision  makers  (Risk);  and,  6)  the  challenge  of  developing  and  gaining 
acceptance  of  analysis  techniques  that  are  not  dependent  upon  traditional 
simulation  or  other  models  (e.g.,  linear  programs)  (Non  Model). 

WG  4  identified  many  recommendations  regarding  new  or  emerging  tools  and 
techniques  that  should  be  explored  and  implemented  where  appropriate.  These 
tools  and  techniques  fell  into  10  categories  as  shown  above.  The  first  category 
listed  involves  standards.  There  are  some  relatively  new  efforts  underway  to 
develop  common  standards  that  can  be  applied  to  the  models  and  data  used  to 
support  analysis;  the  Army  Standards  Category  Coordination  process  is  one 
example  of  this.  DMSO  sponsored  efforts  to  develop  common  conceptual 
models  and  M&S  Resource  Repositories  (MSRR)  is  another  example.  The 
group  recommended  that  the  fundamental  algorithms  underpinning  many  major 
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simulations  in  use  be  examined  in  open  (peer  review)  forums  and  reconciled.  Although 
a  commonly  accepted  set  of  algorithms  would  be  desirable,  it  is  not  necessary  that  all 
services  or  other  users  agree  to  a  single  representation  of  a  warfare  phenomena.  We  can 
agree  to  disagree,  provided  the  differences  are  clearly  documented,  understood  and 
made  available  to  the  analysis  community. 

The  second  technique  identified  by  WG  4  is  automated  data  generation.  The  biggest 
single  complaint  from  the  working  group  was  the  lack  of  needed  data  and  the  difficulty 
and  cost  of  generating  or  obtaining  data  even  when  it  is  available.  Automated  means  of 
generating  commonly  used  data  would  increase  the  responsiveness  and  reduce  the  cost 
of  getting  the  data. 

The  “Analyst’s  Rucksack”  is  an  example  of  using  quick  and  simple  tools  and  decision 
aids  to  fill  in  for  large  model  deficiencies  and  directly  support  warfighters.  The 
Workshop  endorsed  the  development  of  tool  sets  of  this  nature. 

The  commercial  world  is  also  producing  an  array  of  tools  that  have  great  utility  for 
operations  research  analysts.  Tools  which  have  been  around  for  several  years,  but  are 
becoming  increasingly  sophisticated  and  easy  to  use,  include  statistical  and  data 
analysis  packages,  linear  and  nonlinear  programming,  process  simulation  and  decision 
analysis  software.  Emerging  products  include  neural  networks,  Bayes  nets,  colored 
Petri  nets  and  other  software  that  propagate  inferences  or  influences. 

There  is  recent  research  that  is  producing  some  new  techniques  that  may  be  applied  to 
fix,  in  part,  some  of  the  areas  that  lack  tools  for  analysis.  Agent-based  simulation  may 
help  in  the  analysis  of  doctrine,  procedures  and  risk  sensitivity,  and  recent 
mathematical  models  have  been  developed  to  assist  in  the  simulation  (and  real  world 
execution)  of  fusion  between  multiple  sensors  and  sensor  disciplines.  This  is  not  an 
exhaustive  list,  but  are  two  areas  that  show  promise. 

Performing  analysis  in  the  21st  century  will  likely  lead  to  the  resurgence  of  the  multi¬ 
disciplinary  team.  However,  this  team  will  have  a  different  composition  than  the  teams 
in  the  early  days  of  military  OR,  which  were  largely  formed  from  hard  sciences. 
Although  some  problems  may  still  require  physicists,  chemists,  etc.,  most  work  will 
require  groups  pulled  from  3  broad  areas.  The  first  consists  of  domain  experts,  which 
will  often  include  warfighters,  doctrinal  experts,  and  other  expertise  in  military 
operations.  The  second  group  are  software  and  computing  experts,  as  models  and  tools 
become  increasingly  complex  from  a  software  perspective.  The  third  group,  which 
helps  to  meld  the  other  two  together,  consists  of  operations  research  analysts  and  other 
experts  from  related  analysis  domains,  such  as  mathematics  and  statistics.  We  should 
no  longer  expect  that  a  single  military  OR  analyst  be  able  to  read  and  write  code, 
develop  mathematical  models  and  possess  a  broad  military  background  to  include  joint 
operations. 

The  implementation  of  knowledge  management  techniques  identified  by  WG  1  and 
presented  by  plenary  session  and  lunchtime  speakers  will  also  help  address  the 
challenges  to  the  practice  of  OR.  These  concepts  are  developed  in  detail  in  other 
sections  of  this  report. 
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Modeling  and  Simulation  Executive  Agents  (MSEA)  have,  in  some  areas,  greatly 
advanced  the  state  of  the  art  in  M&S.  The  environmental  MSEAs  have  had  a  great  degree 
of  success  and  can  serve  as  exemplars  of  the  process  that  leads  to  significant 
improvements.  They  have  developed  data  standards  applicable  to  a  wide  variety  of 
simulations  (to  include  both  training  and  analysis  simulations)  and  real  world  command 
and  control  systems,  and  populated  these  data  structures  with  data  that  is  easily  obtained 
through  user-friendly  business  processes.  They  have  also  developed  algorithms  to 
transform  the  data  into  information  useful  to  aggregated  models;  for  example,  the  Army 
has  developed  techniques  for  automatically  generating  road  networks  from  detailed 
NATO-standard  mobility  networks. 

One  area  that  can  be  addressed  through  good  knowledge  management  practices  involves 
“other  source”  information.  This  data  includes  NTC  exercise  data,  historical  deployment 
data  and  “lessons  learned”  data.  These  data  sometimes  exist  but  are  not  necessarily 
recorded  for  posterity  (e.g.,  much  NTC  information  is  outbriefed  to  the  unit  commander 
and  then  deliberately  destroyed),  not  captured  at  all  (e.g.,  historical  deployment  data),  or 
are  captured  only  qualitatively  rather  than  quantitatively  (e.g.,  lessons  learned).  These 
data,  if  resources  are  applied  for  capture  and  storage,  can  significantly  improve  our  ability 
to  validate  models  against  “real  world1”  experiences. 

Finally,  the  group  recommended  that  additional  research  be  made  in  the  area  of 
automated  decision  making.  Large  simulation  models  require  some  automated 
representation  of  a  human  decision  maker  (e.g.,  a  commander  and  staff  in  an  analysis 
model  or  an  OPFOR  or  subordinate  commander  in  a  training  model)  to  provide 
reproducibility  or  reducing  exercise/training  costs.  However,  the  current  implementations 
are  recognized  to  be  insufficient  and  unrealistic.  Automated  decision  making  can  also  be 
embedded  in  real  time  decision  aids,  not  to  replace  a  human  decision  maker  but  to  advise 
him  or  her. 


1  We  recognize  that  exercise  data  is  not  real  combat  data,  but  it  is  a  useful  surrogate  that  may  be  more 
accurate  than  outputs  from  abstract  model  and  simulations. 
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Endorsement 


KNOWLEDGE  MANAGEMENT 
HAS  THE  POTENTIAL  TO  ENHANCE 
OPERATIONS  ANALYSIS 

Distinguishes  “what  we  know,  what  we  don’t  know” 
Enhances  Information  Sharing  and  Collaboration 
Addresses  the  “Knowledge  is  Power”  Paradigm 


sms 


While  concepts  associated  with  specific  challenges  were  shown  on  the  last  slide,  the 
Synthesis  Group  also  felt  that  knowledge  management  can  offer  a  more  general  solution 
to  the  broad  issues  confronting  analysis  as  well.  Some  aspects  of  KM  can  already  be 
endorsed  in  ways  that  do  not  threaten  existing  culture.  In  particular,  KM  has  the  ability  to 
help  distinguish  between  what  we  know  and  what  we  don’t  know  by  helping  the 
enterprise  quickly  tap  the  collective  knowledge  of  individuals  in  the  organization.  This 
capability  can  provide  an  individual  assigned  to  work  a  problem  the  capacity  to  quickly 
poll  all  experts  in  or  out  of  the  organization  to  assess  what  is  already  known  or  what  is  in 
the  process  of  being  studied  about  the  problem.  In  fact,  a  properly  implemented  KM 
system  will  save  enormous  time  and  resources  by  helping  the  analyst  and  others  rapidly 
determine  what  is  already  known  as  the  basic  starting  point  of  any  study.  The  same 
system,  by  enabling  rapid  connectivity  between  analytic  practitioners,  will  also  enhance 
information  sharing  and  collaboration.  When  a  study  team,  already  focused  on  a  specific 
area,  realizes  that  the  value  of  its  research  can  be  increased  by  adding  just  a  small  amount 
of  effort  in  order  to  address  a  broader  range  of  questions  to  meet  the  needs  of  a  broader 
customer  base,  one  promise  of  KM  will  be  realized.  And  as  the  limited  resources 
available  in  an  organization  self-organize  to  focus  on  the  areas  they  do  best  in  order  to 
provide  a  broader  base  of  insight  and  information  to  the  organization,  the  people  involved 
will  clearly  understand  that  their  value-added  is  not  in  hoarding  information,  but  rather  in 
developing  and  sharing  it  in  ways  that  enhance  the  overall  capabilities  of  the  enterprise. 
Value  added  will  not  come  from  just  having  information,  since  in  the  future  information 
will  be  increasingly  available  to  everyone;  increased  value  added  will  instead  come  from 
the  ability  of  individuals  and  organizations  to  share  available  information  creatively  to 
enhance  the  capabilities  of  the  organization. 
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Action  Plan 
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-  Working  Group  recommendations  are  included  as  a  backup  to  this  briefing 
and  provide  a  phased  approach  to  KM  and  collaboration  implementation  by 
focusing  on  people,  process,  technology  and  product,  as  well  as  list 
challenges  and  promising  tools  and  methodologies  to  meet  those  challenges 


The  Synthesis  Group  identified  a  number  of  actions,  some  of  which  are  shown 
here,  that  should  become  part  of  a  long  range  action  plan  that  the  community 
should  consider  and  flesh  out  over  the  next  few  years  as  it  evolves  the  practice 
of  military  analysis  in  DoD.  In  particular,  there  are  several  steps  that  MORS 
itself  can  implement  immediately  that  will  have  enormous  potential  to  improve 
the  ability  of  this  community  to  connect  with  each  other  and  better  meet  the 
over  all  needs  of  its  membership.  In  addition  to  the  items  shown  here,  the 
working  groups  provided  a  number  of  general  and  specific  steps  that  will  help 
the  community  learn  to  harness  KM  and  collaboration  as  well  as  to  address 
emerging  issues  in  the  changing  security  environment.  Generalized 
recommendations  endorsed  by  the  Workshop  include  working  the  people, 
process  and  technology  problems  in  order  to  ensure  a  better  final  product. 
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Summary  Recommendations 

♦  Vision/policy:  develop  and  promulgate  enterprise  Vision,  strategic 
directions. 

♦  Culture:  take  action  to  remove  deeply  rooted  cultural  barriers  (e.g.,  dispel 
the  myth  that  “knowledge  is  power”). 

♦  Organization:  Ensure  top  management  buy-in. 

♦  People:  provide  incentives  to  analysts  to  collaborate  and  share  knowledge. 

♦  Processes:  institutionalize  processes  to  enhance  collaboration  and  the 
sharing  of  tacit,  explicit  knowledge  among  analysts. 

♦  Analytic  Tools: 

-  Develop  and  emphasize  joint  standards  in  algorithm  and  data  development. 

-  Take  action  for  developing  and  maintaining  data  warehouses. 

-  Focus  basic  research  and  tool  development  to  meet  emerging  pressures  and 
challenges. 


This  chart  summarizes  the  focus  and  potential  impact  of  the  specific  actions 
recommended.  First,  cultural  changes  are  a  necessary  prerequisite  to  effective 
transformation  of  the  analysis  community  (e.g.,  the  need  to  dispel  the  myth  that 
“knowledge  is  power”).  Second  are  several  organizational  issues  that  must  be 
addressed.  One  of  these  is  to  ensure  top  management  buys  in  to  this 
transformation.  Another  includes  reassessing  how  security  issues  can  be 
addressed  in  an  information  centric,  open  environment.  In  the  third  area, 
decision  makers  must  recognize  that  the  analysts  involved  in  all  phases  of  the 
assessment  process  constitute  a  critical  intellectual  resource.  They  must  be 
given  incentives  to  stimulate  them  to  contribute  and  draw  from  future 
knowledge  warehouses.  Tying  all  this  together  are  the  links  between  the  vision 
and  the  high  level  guidance  to  the  analysis  process  and  the  primary  products  of 
that  process  (i.e.,  effective,  timely,  rigorous  analyses).  To  implement  this 
linkage,  a  set  of  enhanced  processes  is  needed  (e.g.,  institutionalized  processes 
to  diffuse  tacit  and  explicit  knowledge)  supported  by  a  broad  set  of  tools  (e.g., 
knowledge  warehouses,  collaboration  tools).  These  processes  and  tools  are 
constrained  by  available  resources  (e.g.,  funds,  manpower)  such  that  a  special 
funding  mechanism  may  be  required  to  satisfy  time  urgent  joint  needs.  In 
particular,  basic  research  must  be  continued  in  those  areas  where  understanding 
of  phenomenology  is  not  available  (human  decision  processes,  IO/TW,  etc)  as 
well  as  the  continuing  development  of  analysis  tools  to  address  the  emerging 
challenges. 
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Our  Ultimate  Goal 
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The  Synthesis  Group  feels  that  by  addressing  the  needs  of  the  community  to 
implement  our  vision,  the  practice  of  analysis  can  evolve  from  its  current 
position  of  a  limited  toolkit  and  methodologies  to  address  a  limited  number  of 
issues  to  a  complete  tool  set  connecting  analysts  and  decision  makers  to  address 
the  full  array  of  national  security  challenges. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  1  of  7) 


♦  The  DoD  Sponsor  implement  knowledge  management  techniques, 
particularly  to  establish  an  intranet  portal,  in  support  of  an 
important  study  with  suitable  scope,  schedule,  and  content  that 
partners  DoD  components  Military  Operations  Research  Society, 
industry  and  academia.  The  elements  that  should  be  considered 
along  with  strategy  already  discussed  include: 

-  People 

-  Process 

-  Technology 


m§ 


This  is  our  overarching  recommendation,  which  detailed  in  the  next  series  of 
slides  under  the  headings  of: 

Strategy 

People 

Process 

Technology 

Specific  organizations  that  should  be  included  are  the  Services,  OASD(C3I), 
DTIC,  MSIAC,  DMDC,  DMSO. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  2  of  7) 


♦  People 

-  DoD  and  MORS  lead  an  assessment  of  the  capability  to  provide  a 
baseline  inventory  of  analysts  for  DoD  operations  research  studies. 

-  MORS  assess  its  capability  to  establish  a  knowledge  management 
professional  development  program  for  the  DoD  analytic  enterprise 
that  would  include  the  identification  of  skills  and  products,  the 
refinement  of  skill  levels,  and  the  training  of  analysts. 

-  The  DoD  Sponsor  evaluate  successful  skill-inventory  systems  (e.g. 
Northrop-Grumman  and  Y2K)  for  implementation  within  the  DoD 
Analytic  Enterprise. 


m 


Industry  experience  and  DoD  case  studies  show  that  people  is  one  of  the  most 
critical  success  factors  associated  with  KM.  The  recommendations  cited  here 
are  designed  to  provide  the  fundamental  foundation  for  organizing  and 
educating  people  using  KM  techniques  being  applied  by  industry.  Key  DoD 
organizations  that  could  be  involved  in  pursing  this  recommendation  include 
ASD  (C3I),  DUSD(P&R),  OSD(PA&E)  and  DMDC. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  3  of  7) 


♦  Process: 


Skilled 

Analysts 


-  J-8  accept  leadership  with  OASD(C3I)  of  an  evolving  analysis 
process  for  the  purpose  of  improving  analysis  in  a  future  analysis 
state.  Notionally,  the  evolving  process  could  be  closely  interfaced 
with  an  evolving  knowledge  management  process  as  shown  on  the 


Instrumentation 
Plan  for  Data 
Collection 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  4  of  7) 


*  Disseminated  with 
Recommendations 
and  Conclusions 


The  working  group  recognized  that  the  future  (to-be)  process  incorporating  KM 
is  still  to  be  defined.  Nevertheless,  this  slide  helps  illustrate  the  important 
interrelationships  related  to  collaboration,  creation,  and  reuse  of  a  well 
constructed  analytical  KM  environment  and  extensible  knowledge  base. 


Synthesis  WG  -  23 


WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  5  of  7) 


♦  Process  continued: 

-  The  DoD  determine  the  business  rules  for  contributing  analytic 
knowledge.  For  example,  what  are  the  current  rules  that  control 
the  access  to  past  studies  and  how  could  these  rules  be  changed  to 
improve  access  such  as  those  stored  at  Federally  Funded  Research 
and  Development  Centers.  A  special  case  is  the  identification  of 
the  business  rules  for  administering  and  controlling  an  analyst’s 
portal. 

-  The  DoD  Sponsor,  with  the  assistance  of  MORS,  identify  new 
types  of  information  required  for  future  analyses  (e.g.  foreign 
demographics,  information  warfare  related,  economic  information, 
national  infrastructure  related,  chemical/biological,  etc.) 

-  The  DoD  Sponsor  identify  current  Centers  of  Expertise  required 
for  improved  analyses. 


The  working  group  concluded  that  the  first  bullet  was  probably  the  most  critical 
follow-on  task  for  DoD  to  tackle.  If  this  is  not  done,  KM  will  fail. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  6  of  7) 


♦  Process  continued: 

-  Take  steps  to  systematize  the  organization  of  information  by 
transposing  principles  from  traditional  library  and  information 
sciences  to  the  electronic  realm. 

-  Develop  and  use  an  Information  Architecture  that  structures  the 
design  of  information  organization,  labeling,  navigation  and 
indexing  systems  to  support  both  browsing  and  searching. 

-  Identify  and  use  a  set  of  metrics  for  the  business  case  preparation 
and  performance  evaluation  of  the  analyst’s  portal. 

-  Create  knowledge-sharing  incentives  such  as: 

»  Knowledge-sharing  inclusive  performance  reviews 
»  Embedded  authorial  recognition 

H 


The  importance  of  performance  reviews  and  embedded  are  mentioned 
repeatedly  in  the  literature.  The  study  done  by  the  Working  Council  for  the 
Chief  Information  Officers  strongly  supports  this  point.  This  study  also  stressed 
the  importance  of  having  an  information  architecture  for  effective  design  and 
administration  of  browser-enabled  knowledge-based  repositories. 
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WG  #1  Management  of  Analytic  Knowledge 

Recommendations  (Page  7  of  7) 


♦  Technology 

~  The  DoD  concentrate  on  applying  COTS  tools  as  a  key  critical 
success  factor  for  building  analytical  KM  portal.  Example 
technologies  include: 

»  Thesaural  browsers. 

»  Personalized  intranet  portal. 

»  Knowledge-sharing  inclusive  performance  reviews. 

»  Federated  knowledge  management . 

»  Information-expiration  mechanisms. 
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WG  #2  Enabling  Collaboration 

Recommendations 

♦  People 

-  Find  ways  to  identify  (and  evaluate?)  Subject  Matter  Experts 
(SMEs). 

-  Find  a  means  to  develop  operational  experience  in  Civil  Service 
1515s  and  uniformed  analysts  w/o  warfighting  specialties. 

♦  Process 

-  There  is  a  spectrum  of  Collaborative  analysis  processes,  and  each  has 
its  strengths  and  weaknesses  (bold  shifts  versus  consensus). 

»  Can  organize  for  competitive  analyses. 

»  Can  organize  to  search  for  a  “common  ground”  solution. 

-  Collaborative  Analysis  is  a  “team  sport”  and  requires  practice. 

»  OA  Support  to  Commanders  in  the  field  most  effective  when 
“Train  as  you  Fight”  has  been  implemented. 


Our  recommendations  have  been  collected  under  the  headings  of  “People,” 
“Process”  and  “Product.” 


Synthesis  WG  -  27 


WG  #2  Enabling  Collaboration 

Recommendations 

♦  Process  (continued) 

-  Reestablish  formal  liaison/exchanges  to  improve  the  quality  of 
collaborative  analysis 

-  Support  independent,  accredited  sources  for  data  to  speed 
collaborative  effort. 

-  Find  ways  to  identify  data  to  source  new  applications  and  phenomena. 

-  Data  collection  must  be  deliberate  and  done  in  such  a  way  that  it  does 
not  require  additional  effort  by  operators. 

-  Synchronize  data  production  with  data  demands  and  institutionalize 
production  and  update  of  long-lead  items. 

»  E.g.,  extend  JMEM  to  include  developmental  systems. 

-  Harmonize  schedule  of  major  collaborative  projects  and  Service  workload. 
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WG  #2  Enabling  Collaboration 

Recommendations 


♦  Product 

-  Need  improvements  in  literature  search  capabilities,  identifying 
work  in  progress  that  may  affect  the  project  at  hand. 

-  Study  documentation  techniques  must  also  improve,  allowing  for 
reproducibility  of  results  and  reuse  of  data  in  addition  to 
description  of  procedure  and  findings. 
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WG  #3  New  Analysis  Challenges 

Recommendations 


♦  Find  methods  to  deal  with. 

-  New  forms  of  operations. 

-  Traceability  of  known  factors  and  interactions. 

-  Quick  turn-around  studies. 

♦  Nurture  analysts  and  analytical  thinking  rather  than  focusing  on 
model  development. 

-  Project  more  balanced  view  of  model  capabilities  and  limitations. 

-  Reverse  trend  to  big  models  as  solution  to  all  problems. 

♦  Characterize  and  communicate  impact  of  uncertainties  (e.g., 
Risk,  behavioral  factors). 


In  summary,  as  the  analytical  community  evolves  it  needs  to  find  tools  to  deal 
with  new  forms  of  operations,  find  methods  to  trace  known  factors  and 
interactions  to  their  logical  combat  output  and  find  tools  that  will  provide 
accurate  and  thorough  analysis  in  a  short  timeline. 

The  DoD  analytical  community  needs  to  nurture  analysts  and  emphasize 
analytical  thinking  rather  than  focusing  on  simulation  and  modeling 
development.  The  community  must  project  a  more  balanced  view  of  model 
capabilities  as  well  as  limitations,  and  reverse  the  recent  trend  toward  blind  faith 
toward  large  models  as  the  solution  to  all  analytical  problems. 

The  analytical  community  needs  to  find  means  for  characterizing  and 
communicating  the  impacts  of  uncertainty.  For  example  risk,  such  as  decision 
risk  based  on  the  fact  that  various  factors  may  not  have  been  considered  in  the 
current  analysis,  need  to  be  characterized  and  communicated  to  the  decision 
maker, the  analytical  product  customer. 
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WG  #4  New  Analysis  Tools  and  Methods 

Near  Term  Recommendations  (1  of  3) 


♦  Increase  DoD  emphasis  on  supporting  inter-service  collaboration 
toward  standards. 

-  This  builds  on  the  existing  service  efforts. 

-  Possible  Organizational  Models  Are: 

»  Existing  JTCG  (JMEM’s)  is  a  possible  model. 

»  MSRR  Board  of  Directors. 

-  This  should  include  Joint  rules  for  reconciliation  of  standards. 

-  Establish  a  Baseline  (i.e.  a  minimal  standard  for  meta-data). 

♦  Use  limited  experience  as  a  foundation  for  KM  implementation. 

-  Formal  plan  for  achieving  the  vision. 

»  Senior  Leader  Support  and  involvement. 

»  Deliberate  Investment  Strategy 
»  Uses  the  Multi-disciplinary  team. 

»  There  should  be  a  co-evolution  of  culture  and  funding  for 


technology. 
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WG  #4  New  Analysis  Tools  and  Methods 

Near  Term  Recommendations  (2  of  3) 


♦  The  Analysis  community  should  be  represented  in  the  DII  COE 
process. 

♦  The  DoD  should  develop  and  resource  a  strategy  for  collecting 
“other  source”  data  and  applying  KM. 

-  NTC  exercise  data. 

-  Historical  deployment  data. 

-  Lessons  learned  data. 

-  etc. 


maia 
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WG  #4  New  Analysis  Tools  and  Methods 

Near  Term  Recommendations  (3  of  3) 

♦  Encourage  the  creation  and  use  of  “quick  and  dirty”  tools.  This 

should  be  accompanied  by: 

-  Rules  of  the  road  for  using  data. 

-  Best  Practices. 

-  Mechanisms  for: 

1/ 

»  Timely  peer  review. 

»  Documenting  use. 

\  Knowledge 

»  Allowing  evolution. 

Management  A1 

GEH 
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WG  #4  New  Analysis  Tools  and  Methods 

Recommendations  for  Future  Research 

♦  A  detailed  review  should  be  conducted  that  maps  ongoing  research 
to  identified  shortcomings. 

-  Advanced  Concepts  that  “automate”  human  decision-making  (C2). 

-  Intelligent  agents. 

-  Inference/fusion. 

-  Visualization. 

-  etc. 

♦  An  investment  strategy  should  then  be  developed  to  fill  the 
identified  holes. 
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TERMS  OF  REFERENCE 

Evolving  the  Practice  of  Military  Operations  Analysis 
in  DoD  Applying  Knowledge  Management 


1.  Background 

Leaders  in  the  DoD  analysis  community  are  calling  for  an  examination  of  how  we  perform  analyses. 
They  are  asking  us  whether  our  analysis  practices  warrant  examination  with  a  view  towards  improving  their 
effectiveness. 

The  management  of  knowledge,  how  to  prepare  to  perform  the  next  analysis,  how  to  perform  the  next 
analysis,  how  to  gather  information  needed  to  support  analysis,  and  how  to  find  and  apply  analytic  insights 
from  recent  studies,  is  becoming  a  principal  factor  in  being  able  to  meet  decisionmakers’  expectations  in 
regard  to  both  the  quality  of  an  analysis  and  the  speed  with  which  an  analysis  is  accomplished. 

Making  Analysis  Relevant1  was  the  topic  of  a  PHALANX  article  by  Mr  Vince  Roske  in  early  1998.  Mr 
Roske  reiterated  a  recipe  provided  by  John  D.  (Dave)  Robinson,  MG  USA,  Ret,  when  he  was  the  Director  of 
J~8.  The  recipe  is  provided  below. 

•  What's  the  question? 

•  What's  the  “real”  question? 

•  What  do  the  final  slides  look  like? 

•  What  do  1  already  know ? 

•  How  do  I  get  the  remaining  information  that  I  need? 

Dave  Robinson’s  recipe  is  reminiscent  of  the  very  practical  battle  drills  used  by  the  military  to  train 
soldiers  to  do  basic  tasks.  It  is  simple,  it  is  memorable,  and  it  will  get  the  job  done.  Mr  Roske  goes  on  to 
look  deeper  at  each  of  the  recipe  steps  and  discusses  the  issues  associated  with  performing  analysis  today. 
He  explains  that  we  must  focus  on  the  purpose  of  the  analysis  and  not  become  wed  to  models  that  don’t  fit 
the  analysis.  Furthermore,  he  explains  what  the  analyst  must  do  to  keep  the  analysis  relevant  to  the  decision 
maker. 

The  Revolution  in  Analytical  Affairs2  is  described  and  discussed  by  LTG  Dave  Heebner  in  his  excellent 
PHALANX  article  published  later  in  1998.  He  emphasizes  many  of  the  same  points  that  Mr  Roske  focused 
on.  In  particular  he  cites  the  need  to  pin  down  the  decisionmaker’s  questions  early,  and  provide  the  needed 
findings,  recommendations,  and  insights  as  quickly  as  possible. 

2.  Discussion 

The  first  three  steps  in  General  Robinson’s  recipe  address  critical  steps  in  organizing  and  focusing  an 
analysis  or  study.  Answering  “ What's  the  question?”  and  “What's  the  “real”  question?”  enables  the 
analyst  to  frame  a  study  so  that  there  is  a  chance  that  it  may  ultimately  be  relevant  to  the  needs  of  the 
decision  maker.  Answering  “ What  do  the  final  slides  look  like?”  up  front  serves  the  dual  purpose  of 
stressing  the  importance  of  communicating  results  to  the  decisionmaker  and  focusing  the  analysis  efforts 
from  the  beginning. 


1  Mr  Vincent  P.  Roske,  Jr.,  “Making  Analysis  Relevant,”  PHALANX  Volume  31,  Number  1,  March  1998 

2  LTG  David  K.  Heebner  USA,  “Revolution  in  Analytical  Affairs,”  PHALANX  Volume  31,  Number  3,  September  1998 
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But  the  substantive  content  of  the  analysis  is  contained  in  the  last  two  recipe  steps:  “What  do  I  already 
know?”  and  “How  do  I  get  the  remaining  information  I  need?”  Evolving  the  Practice  of  Military  Operations 
Analysis  in  DoD  must  therefore  substantively  address  these  last  two  questions. 

One  of  the  keys  to  successfully  evolving  the  practice  of  analysis  is  the  innovative  management  and 
application  of  knowledge,  both  tacit  (the  knowledge  in  the  minds  of  the  experts)  and  explicit  (the  knowledge 
that  is  recorded  in  various  ways).  There  is  an  explosion  of  new  thinking  and  new  supporting  technology  to 
capture  and  apply  knowledge  more  effectively.  This  area  includes  innovation  in  the  discovery,  expansion, 
capture,  organization,  and  sharing  of  knowledge.  Innovations  include:  correlation  landscapes,  data  mining 
tools,  virtual  information  centers  and  influence  diagrams 

For  a  discussion  of  the  emerging  field  of  Knowledge  Management  see  Appendix  A. 

For  a  bibliography  of  Knowledge  Management  information  sources  see  Appendix  B. 

The  keys  to  effective  analysis  in  DoD  are  the  analysts,  the  knowledge  that  analysts  access,  the  process 
by  which  analysts  perform  their  work,  the  tools  that  analysts  employ  to  perform  their  work,  and  the 
effectiveness  with  which  analysts  communicate  analytical  results  to  decisionmakers.  Analysts  and  subject 
matter  experts  have  important  and  relevant  knowledge  in  their  heads  -  tacit  knowledge  -  that  could  spark 
important  insights  among  their  peers  if  it  were  available  to  them. 

Information  is  a  source  for  good  decisionmaking,  however,  a  piece  of  information  is  more  valuable 
when  presented  in  context,  that  is,  with  other  pieces  of  related  information  that  make  the  original 
information  more  meaningful  and  actionable.  For  instance,  a  study  of  the  effectiveness  of  a  proposed  ISR 
system  may  provide  important  information,  but  it  is  much  more  meaningful  and  actionable  when  analyzed  in 
the  context  of  national  security  policy  and  DoD  objectives  and  when  related  to  operational  outcomes. 

In  DoD,  we  analysts  don’t  know  what  we  know,  and  consequently,  we  are  bound  to  reinvent  the  wheel 
many  times.  We  do  not  have  automated  enterprise  repositories  which  hold  accessible,  important,  explicit 
knowledge  nor  do  we  have  the  means  to  locate  the  right  subject  matter  experts  quickly  and  easily  to  access 
their  tacit  knowledge. 

The  proper  management  of  knowledge  can  improve  analytical  efficiency  by  providing  a  framework, 
tools,  and  techniques  to  reuse  intellectual  assets.  By  marshalling  resources  to  respond  to  opportunities  and 
threats,  our  responsiveness  could  be  vastly  improved.  And  by  bringing  people  together  across  time  and 
geography  to  share  ideas  and  insight,  intellectual  innovation  can  flourish  and  bear  richer  insight  for  decision 
makers. 

This  special  meeting,  comprising  analysts  and  subject  matter  experts  in  approaches  to  analysis,  will 
concentrate  on  the  improvement  of  the  analysis  practice  throughout  DoD.  To  achieve  this  end,  participants 
will  enumerate  potential  issues  related  to  a  broad  move  towards  more  collaborative  analysis  and  enterprise 
knowledge  management  and  define  the  appropriate  policies,  organizational  structure,  processes, 
technologies  and  training  needed  to  institutionalize  an  improved  practice  of  analysis  in  DoD.  These  issues 
will  be  addressed  by  working  groups,  addressing  the  spectrum  of  dimensions  of  organizational  change,  as 
well  as  supporting  infrastructures  and  supporting  tools  and  techniques. 

3.  Goals  and  Objectives 

The  body  of  analytical  methods,  techniques,  and  tools  and  how  we  apply  them  will  be  the  focus  of  this 
workshop.  We  will  be  examining  how  we  carry  out  our  professional  work  and  the  need  to  evolve. 

Questions  to  be  addressed  include: 

•  What  should  be  the  customary  manner  for  performing  defense-related  analyses? 

•  How  do  we  manage  the  knowledge  we  derive  from  our  analyses? 

•  How  do  we  support  both  fast  turn-around  and  more  deliberate  analyses? 
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•  What  problems  are  not  well  served  by  the  current  state  of  the  practice? 

•  What  innovative  developments  in  analysis  tools  and  methods,  including  Knowledge 
Management,  hold  potential  value  for  DoD  analysis? 

3.1  Goals 

This  workshop  will  afford  the  military  OR  community  an  opportunity  to  achieve  the  following  goals: 

(1)  Develop  a  shared  understanding  of  the  need  to  evolve  the  practice  of  Military  Operations 
Analysis  in  DoD; 

(2)  Establish  a  vision  for  improving  analytic  practice  and  the  management  of  supporting  analytic 
knowledge  in  DoD; 

(3)  Create  a  plan  to  realize  the  vision  to  improve  analytic  practice  and  the  management  of 
supporting  analytic  knowledge  in  DoD;  and 

(4)  Define  how  MORS  may  work  with  the  sponsors  over  time  to  implement  improvements  to 
analytic  practice  and  to  the  management  of  analytic  knowledge  in  DoD. 

3.2  Objectives 

The  objectives  are  to: 

•  Define  a  desired  analytic  practice  approach  in  DoD  in  terms  of  policy,  organization,  process, 
technology,  and  training. 

•  Define  current  analytic  practice  in  DoD  in  terms  of  policy,  organization,  process,  technology, 
and  training. 

•  Identify  weaknesses  in  the  current  practice  relative  to  current  and  future  requirements. 

•  Produce  an  actionable  plan  to  move  to  the  desired  state. 

•  Define  mechanisms  to  be  established  within  MORS  to  work  with  the  DoD  sponsors  to  execute 
the  plan. 

4.  Approach 

In  order  to  achieve  these  goals  and  objectives,  the  subject  of  analytic  practice  will  be  examined  in  a 
number  of  different  contexts.  Working  groups  will  examine  analytic  practice  within  each  contextual 
framework.  Working  groups  will  be  asked  to  address  some  baseline  issues  (as  a  starting  point)  and  provide 
specific  results.  The  working  group  contexts,  baseline  issues,  and  desired  results  are  detailed  below. 

4.1  Working  Groups 

The  workshop  attendees  will  be  organized  into  four  domain  working  groups  (25  to  30  members  each) 
plus  a  synthesis  group: 

•  Management  of  Analytic  Knowledge 

•  Examine  the  manner  in  which  analytic  knowledge  is  discovered,  expanded,  organized,  and 
shared. 

•  Enabling  Collaboration 

•  Examine  the  process,  tools,  and  techniques  that  support  distributed  analytic  teams  in 
performing  analytic  tasks. 

•  New  Analysis  Challenges 
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•  Examine  military  operations  analysis  questions  of  interest  for  which  the  established  military 
operations  analysis  community  apparently  does  not  have  answers. 

•  New  Analysis  Tools  and  Methods 

•  Examine  emerging  developments  in  analytical  approaches  to  complex  systems  to  assess 
potential  applications  to  challenging  military  operations  analysis  probleMs 

•  Synthesis 

•  Discover  the  common  themes  that  tie  together  the  efforts  of  the  four  subject  area  working 
groups  and  provide  feedback  to  the  groups  on  a  continuing  basis. 

4.1.1  Working  Group  1:  Management  of  Analytic  Knowledge  Working  Group 

Chair:  Mr  Mike  Yoemans 

Co-Chair:  MAJ  Willie  McFadden 

This  working  group  will  examine  the  literature  that  is  beginning  to  define  the  field  of  knowledge 
management.  The  group  will  examine  the  manner  in  which  analytic  knowledge  is  discovered,  expanded, 
organized,  and  shared.  Also  addressed  will  be  intelligent  search  engine  technology  and  intelligent  agent 
technology  and  how  these  techniques  leverage  web-based  stored  data  to  meet  real-time  analysis 
requirements.  The  group  will  produce  a  summary  report  defining  how  these  technologies  may  be 
incorporated  into  existing  analytic  practice. 

4.1.2  Working  Group  2:  Enabling  Collaboration  Working  Group 

Chair:  COL  Bob  Clemence 

Co-Chair:  Lt  Col  Kirk  Yost 

This  working  group  will  examine  collaboration  techniques.  Particular  attention  will  be  paid  to  policy, 
procedure,  and  technique  to  support  collaboration  once  expertise  has  been  identified.  While  efforts  to 
promote  collaboration  are  ongoing,  there  are  still  opportunities  to  improve  the  sharing  of  information, 
expertise,  and  insights.  Early  techniques  supporting  collaboration  are  maturing  and  new  techniques  are 
emerging.  The  group  will  also  examine  expertise  location  techniques.  Particular  attention  will  be  paid  to 
policy,  procedure,  and  technique  to  enable  the  development  and  maintenance  of  a  DoD-wide  means  to  find, 
catalog,  and  make  available  analysis  expertise  and  subject  matter  expertise.  Attention  should  be  paid  to  the 
dilemma  organizations  face  in  gaining  the  willing  cooperation  of  its  most  knowledgeable  members  in 
seeking  organizational  improvement.  The  group  will  produce  a  summary  report  defining  how  expert 
knowledge  may  be  shared  to  support  analytic  practice. 

4.1.3  Working  Group  3:  New  Analysis  Challenges 

Chair:  Dr  Steve  Pilnick 

Co- Chair:  Maj  Suzanne  Beers 

This  working  group  is  about  new  problems,  problems  not  involving  conventional  warfare,  and/or  post- 
Cold  War  probleMs  Actually  there  are  old  problems  that  belong  here  as  well.  The  issue  isn’t  whether  the 
problem  is  new  or  old,  but  rather  whether  the  Military  OR  community  has  succeeded  in  addressing  it 
satisfactorily.  An  example  is  the  problem  of  determining  the  value  of  a  common  operating  picture  to  the 
warfighter.  Presentations  in  this  working  group  should  come  from  the  major  analysis  players  like  PA&E 
and  the  Services’  Studies  and  Analysis  shops,  from  the  warfighters  like  Joint  Force  Command  and 
CINCPAC,  and  from  decision  makers  like  OSD,  J8,  and  the  Services’  warfare  requirements  and  assessment 
organizations.  The  group  will  produce  a  summary  report  describing  the  analytic  problem  domains  explored, 
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the  problem  domains  that  may  be  addressed  with  known  techniques,  and  the  problem  domains  that  might  be 
well-served  by  the  development  of  new  analytic  techniques.  This  working  group  will  consult  with  Working 
Group  4  throughout  the  Workshop  to  ensure  that  new  analysis  challenges  and  new  analysis  tools  and 
methods  are  shared  between  the  two  working  groups  as  their  work  proceeds. 

4.1.4  Working  Group  4:  New  Analysis  Tools  and  Methods 

Chair:  LTC  Dan  Maxwell 

Co-Chair:  Dr  A1  Brandstein 

This  working  group  should  hear  about  chaos,  complexity  theory,  complex  adaptive  systems,  agent-based 
modeling,  and  other  emerging  methods  for  analyzing  complex  systeMs  The  working  group  should 
brainstorm  potential  Military  Operations  Analysis  in  DoD  applications  of  each  new  approach  considered  and 
attempt  to  assess  the  potential  of  such  new  methods  to  deal  with  analysis  challenges  identified  in  Working 
Group  3.  The  group  will  produce  a  summary  report  describing  the  classes  of  tools  and  methods  explored, 
and  the  Working  Group’s  ideas  about  the  classes  of  problems  that  may  be  addressed  with  these  tools  and 
methods.  The  Working  Group  report  should  be  useful  as  an  outline  of  a  research  agenda  for  next  generation 
military  OR.  This  working  group  will  consult  with  Working  Group  3  throughout  the  Workshop  to  ensure 
that  new  analysis  challenges  and  new  analysis  tools  and  methods  are  shared  between  the  two  working 
groups  as  their  work  proceeds. 

4.1.5  Synthesis  Group 

Chair:  Dr  Stuart  Starr,  FS 

Co-Chair:  Dr  Jerry  Kotchka 

This  group  will  take  a  broad  view,  identifying  the  high  level  issues  across  the  domains  of  policy, 
organization,  process,  technology,  and  training  that  enable  or  impede  the  successful  evolution  of  both 
analytic  practice  and  knowledge  management  in  the  MORS  community.  The  Synthesis  Group  will  provide 
the  integrating  perspective. 

4.2  Working  Group  Tasking 

Working  groups  will  be  directed  to  address  the  following  questions  and  concerns  in  the  following 
priority  order: 

To  what  degree  and  in  what  terms  may  the  working  group’s  focus  area  be  defined? 

How  may  the  working  group’s  area  of  focus  be  addressed  to  meet  current  and  future  needs? 

Working  groups  will  define  up  to  three  priority  issues  that  they  feel  must  be  addressed  to  enable  their 
focus  area  to  be  improved.  Each  priority  issue  will  be  defined  along  with  a  summary  action  plan 
recommending  how  the  issue  may  be  addressed.  The  summary  action  plan  will  recommend  an  action 
organization,  supporting  organizations,  and  principal  steps  to  action  resolution. 

5.  Sequence  of  Events 

A  working  group  chair  and  co-chair  warm-up  session  will  be  held  the  evening  before  the  first  day  of  the 
workshop.  This  session  will  be  held  at  the  workshop  hotel.  The  purpose  of  this  warm-up  session  is  to 
review  and  discuss  the  tasking  for  the  working  groups. 

The  morning  of  the  first  day  will  be  devoted  to  a  plenary  session.  The  Workshop  Chair  will  present  the 
goals,  objectives,  and  organization  for  the  workshop.  Mr  Roske  will  present  the  charge  to  the  workshop 
participants.  The  plenary  session  speakers  will  provide  alternative  perspectives  on  analytical  practice  in 
DoD  and  knowledge  management  in  DoD.  The  first  keynote  speaker  will  address  the  emerging  concept  of 
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enterprise  knowledge  management  and  how  it  enables  improved  decisionmaking.  The  second  and  third 
keynote  speakers  will  provide  the  decisionmakers’ requirement  for  improved  analysis. 

There  will  be  a  social  event  (mixer)  at  the  end  of  the  first  day. 

There  will  be  a  luncheon  speaker  on  the  first  and  second  day. 

The  afternoon  of  the  first  day  and  all  of  the  second  day  will  be  devoted  to  working  group  sessions  and 
discussions.  The  morning  of  the  third  day  will  be  devoted  to  the  preparation  of  working  group  presentations. 
The  afternoon  of  the  third  day  will  be  devoted  to  working  group  presentations  in  plenary  session. 

The  morning  of  the  fourth  day  will  be  made  available  to  working  group  chairs  and  co-chairs  for  final 
editing  of  the  working  group  annotated  briefings. 


6.  Agenda 

Day/Time 

Activity 

POC 

Location 

Monday 

February  28,  2000 

1900 

Working  Group  Co-Chair  Warm-Up  Session 

Mr  Denis  Clements 

Hotel 

Tuesday 

February  29,  2000 

0700 

Registration  and  Continental  Breakfast 

MORS  Office 

NPS 

0800 

Call  to  Order  and  Opening  Remarks 

Mr  Denis  Clements 

NPS 

0805 

Welcome  by  Host 

Prof.  Dave  Schrady 

NPS 

0810 

Sponsor’s  Welcome  and  Workshop  Kick-Off 

Mr  Vince  Roske 

NPS 

0830 

Keynote:  "Enterprise  Knowledge  Management" 

Dr  Ruth  David 

NPS 

0930 

Break 

NPS 

0945 

Speaker:  "Perspectives  on  Requirements 
Analysis" 

Mr  Jim  Johnson 

NPS 

1045 

Speaker:  "Perspectives  on  Capabilities 

VADM  Conrad 

NPS 

Analysis" 

Lautenbacher 

1145 

Charge  to  the  Working  Groups 

Mr  Denis  Clements 

NPS 

1200 

Lunch  with  Guest  Speaker: 

MORS  Office,  Speaker 

NPS 

1330 

Working  Group  Session  #1 

Working  Group  Co-Chairs 

NPS 

1510 

Break 

MORS  Office 

NPS 

1520 

Working  Group  Session  #2 

Working  Group  Co-Chairs 

NPS 

1700 

Adjourn  Regular  Session 

Mr  Denis  Clements 

NPS 

1700 

WG  Chair  and  Co-Chair  Hot  Wash 

Mr  Denis  Clements 

NPS 

1715-1900 

Mixer 

MORS  Office 

NPS 

Wednesday 

March  1,  2000 

0715 

Continental  Breakfast 

MORS  Office 

NPS 

0800 

Plenary  Session:  "QDR  Workshop  Report" 

Mr  Mike  Leonard 

NPS 

0900 

Working  Group  Session  #3 

Working  Group  Co-Chairs 

NPS 

1015 

Break 

MORS  Office 

NPS 

1030 

Working  Group  Session  #4 

Working  Group  Co-Chairs 

NPS 

1200 

Lunch  With  Guest  Speaker: 

MORS  Office,  Speaker 

NPS 

1330 

Working  Group  Session  #5 

Working  Group  Co-Chairs 

NPS 

1515 

Break 

MORS  Office 

NPS 

1530 

Working  Group  Session  #6 

Working  Group  Co-Chairs 

NPS 

1700 

Adjourn  Regular  Session 

Mr  Denis  Clements 

NPS 

1700 

WG  Chair  and  Co-Chair  Hot  Wash 

Mr  Denis  Clements 

NPS 

Thursday 

March  2,  2000 

0715 

Continental  Breakfast 

MORS  Office 

NPS 

0800 

Working  Groups:  Prepare  Briefings 

Working  Group  Co-Chairs 

NPS 

0945 

Break 

MORS  Office 

NPS 

1000 

Working  Groups:  Prepare  Briefings 

Working  Group  Co-Chairs 

NPS 
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1230 

Lunch 

MORS  Office 

NPS 

1330 

Working  Groups:  Present  Briefings,  WG  1,  2,  3 

Working  Group  Co-Chairs 

NPS 

1500 

Break 

MORS  Office 

NPS 

1515 

Working  Groups:  Present  Briefings,  WG  4  and 
Synthesis  Group 

Working  Group  Co-Chairs 

NPS 

1615 

Workshop  Wrap-Up 

Mr  Denis  Clements 

NPS 

1630 

Adjourn 

Mr  Denis  Clements 

NPS 

Friday 

March  3,  2000 

0700 

Working  Group  Co-Chairs  complete  Working 
Group  Annotated  Briefings 

Working  Group  Co-Chairs 

NPS 

1100 

Adjourn  Workshop 

Mr  Denis  Clements 

NPS 

7.  Attendees 

Attendance  will  be  controlled  via  invitation.  Attendees  will  include  invited  experts  from  OSD,  all 
Services,  the  Joint  Staff,  Federally  Funded  Research  and  Development  Centers,  operational  commanders, 
and  DoD  contractors.  Workshop  chairs  will  control  membership  of  their  sessions  in  conjunction  with  the 
Organizing  Committee.  Attendance  will  be  limited  to  120  people. 

8.  Products 

There  will  be  four  specific  products  generated  as  a  result  of  this  workshop: 

•  An  Executive  Summary  in  the  form  of  a  text  document  and  a  scripted  briefing  for  the  MORS 
Sponsors  addressing  the  workshop  objectives,  findings,  conclusions,  and  recommendations. 

•  A  proceedings  containing  summaries  of  all  sessions  and  copies  of  appropriate  briefing  slides 
and  presentations. 

•  A  general  session  presentation  for  the  68th  MORSS. 

•  A  PHALANX  article 


9.  Milestone  Plan 


Action 

whom 

Due 

Pre-Workshop  Planning  Milestones 

Initiate  draft  TOR  procedure 

Initiator 

06/04/99 

Select  tentative  dates 

Initiator,  VPMO 

EVP,  VPA 

08/03/99 

Provide  ’’For  Comment"  draft  of  TOR  to  Sponsors,  EC 

MORS  Office 

09/02/99 

Revise  TOR 

Initiator  & 

MORS  Office 

10/02/99 

Circulate  initial  draft  TOR  to  MORS  office  and  committee  for 

comments 

Chair 

10/12/99 

Solicit  candidates  for  working  group  chairs 

Chair 

10/12/99 

Circulate  final  draft  TOR  to  MORS  office  and  proponent(s)  for 
concurrence 

MORS  Office 

11/01/99 

Select,  Invite  Plenary  Speakers 

Chair 

11/22/99 

Select  working  group  chairs 

Chair 

12/01/99 

Provide  read-ahead  materials  and  releases  to  MORS  Office 

Committee 

12/01/00 

Approve  TOR,  Program  Chair,  budget  and  fees 

EC 

12/06/99 

Select  organizations  to  be  invited  and  prepare  letter  inviting 
nominations. 

Chair/Co-Chair 

12/10/99  J 

Post  read-ahead  materials  on  MORS  web  page 

MORS  Office 

12/13/99 
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Mail  invitations  to  nominate  attendees 

MORS  Office 

12/15/99 

Submit  nominations  or  requests  for  applications 

Nominating  organizations 

12/30/99 

Pre-registration,  security  clearances  due  at  MORS  Office 

Invitees 

01/14/00 

Select  invitees 

Committee 

01/21/00 

Mail  invitations,  WG  assignments 

MORS  Office 

02/07/00 

MORS  Staff  departs  for  NPS 

MORS  Office 

2/26/00 

Working  Group  Warm-Up  Session 

Chair/Co-Chair 

2/28/00 

Post-Workshop  Planning  Milestones 

Complete  Annotated  Briefings 

WG  Chairs 

03/03/00 

Submit  PHALANX  Article 

Chair/Co-Chair 

03/15/00 

Brief  sponsors 

Chair 

03/30/00 

Submit  After- Action  Report 

Chair 

04/28/00 

Complete  written  products 

Committee 

05/30/00 

68th  MORSS  Presentation 

Chair 

06/20-22/00 

Approve  written  products 

Pubs  Committee 

06/28/00 

Review  approved  products 

Proponent(s) 

07/28/00 

Distribute  approved  products 

MORS  Office 

08/28/00 

Organizing  Committee  Meetings 

Organizing  Committee  Meeting- 1 

Organizing  Committee 

10/27/99 

Organizing  Committee  Meeting-2 

Organizing  Committee 

11/09/99 

Organizing  Committee  Meeting-3 

Organizing  Committee 

11/22/99 

Organizing  Committee  Meeting-4 

Organizing  Committee 

12/08/99 

Organizing  Committee  Meeting-5 

Organizing  Committee 

01/04/00 

Organizing  Committee  Meeting-6 

Organizing  Committee 

01/18/00 

Organizing  Committee  Meeting-7 

Organizing  Committee 

02/03/00 

Organizing  Committee  Meeting-8 

Organizing  Committee 

02/15/00 

10.  Proponents  for  the  Workshop  Are: 


•  Army 

•  Navy 

•  Air  Force 

•  Marine  Corps 

•  OSD 

•  Joint  Staff 


11.  Planning  and  Organizing  Chairs  and  Committees 


Lead  Proponent: 

Chair: 

Co-  Chair: 

Synthesis  Chair: 

NPS  Site  Coordinator: 
Administrative  Coordinators: 


Organizing  Committee: 


Mr  Vince  Roske 

Mr  Denis  Clements 

Ms  Sue  Iwanski 

Dr  Stuart  Starr 

CDR  Matt  Boensel 

Richard  Wiles  and  Natalie  Kelly 

MORS,  Suite  202 

101  S.  Whiting  Street 

Alexandria,  VA  22304 

(703)  751-7290 

Army,  Dr  Bob  Hinkle,  MAJ  Bruce  Simpson 
Navy,  Mr  Chuck  Werchado 
Air  Force,  Mr  Clay  Thomas,  Lt  Col  Kirk  Yost 
Marine  Corps,  Capt  Russ  Bergeman 
OSD,  LTC  Dan  Maxwell 
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Joint  Staff,  COL  Bob  Clemence 
AFSPACECOM,  Maj  Suzanne  Beers 
NPS,  Dr  Dave  Schrady,  Dr  Steve  Pilnick 
Institutional  Members:  IDA,  Dr  Tom  Allen 

MITRE,  Dr  Stuart  Starr 
LMCO,  Dr  Jerry  Kotchka 
SAIC,  Ms  Annie  Patenaude 

12.  Administration 

Name:  Ms  Natalie  Kelly 

Dates:  February  28,  2000  through  March  3,  2000 

Location:  Naval  Postgraduate  School,  Monterey,  CA 

Fee:  Federal  Government  Employees  $190;  All  others  $380 

Attendance:  Limited  to  120 

Classification:  UNCLASSIFIED 

Hotel:  Hyatt  Regency,  Monterey,  #1  Old  Golf  Course  Road,  Monterey,  CA  93940,  831-372-1234,  Ask  for 
the  MORS  Block  of  RooMs 
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APPENDIX  A:  DESCRIPTION  OF  KNOWLEDGE  MANAGEMENT 


The  three  essential  aspects  of  knowledge  management  are  knowledge  discovery,  expansion,  and  capture, 
knowledge  organization,  and  knowledge  sharing,  see  Figure  1.  We  discover  knowledge  where  it  is:  in  the 
heads  of  people,  in  manuals  and  reports,  and  mined  from  every  node  of  the  Internet  and  the  SIPRNET.  We 
must  expand  knowledge  through  research  and  development,  by  examining  the  problems  for  which  we  do  not 
have  the  techniques  and  tools  in  place.  We  must  capture  the  knowledge  that  we  discover  and  create.  We 
must  organize  the  knowledge  according  to  a  preferred  scheme  or  taxonomy  and  we  must  share  that 
knowledge  among  the  analysts  and  subject  matter  experts  so  we  may  derive  benefit  from  it. 


Figure  1.  The  Essential  Components  of  Enterprise  Knowledge  Management 

The  discovery,  expansion,  capture,  organization,  and  sharing  of  knowledge  can  be  accomplished  by 
leveraging  current  initiatives  and  disciplines.  Starting  with  the  wealth  of  information  and  data  currently 
available  in  manuals,  reports,  the  GCCS  and  GTN  data  stores,  and  automated  open  sources,  we  can  leverage 
our  tacit  and  explicit  knowledge  by  building  a  knowledge  management  structure  on  top  of  six  key  pillars: 
advanced  data  analysis,  knowledge  discovery  and  mapping,  Knowledge  expansion,  expertise  location, 
collaboration,  and  knowledge  transfer,  see  Figure  2. 

Advanced  data  analysis  is  much  more  than  the  traditional  data  analysis.  It  describes  the  processes, 
tools,  and  techniques  that,  when  combined,  are  used  to  enable  improved  decisionmaking.  Data  analysis 
must  expand  to  include  data  mining,  on-line  analysis  processing  (OLAP),  correlation  landscapes,  influence 
diagrams,  and  other  advanced  technologies  that  can  be  used  for  gleaning  valuable  insights  from  stored  data. 

Knowledge  discovery  includes  text  mining  techniques  that  enable  knowledge  discovery  from  text 
sources.  Knowledge  mapping  is  the  technique  for  representing  knowledge  sources  (people  and  information 
nodes)  in  a  context  defined  by  their  relationships. 
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Figure  2.  The  Pillars  of  Knowledge  Management 

Knowledge  expansion  depends  on  the  examination  of  current  and  future  analytical  need.  As  these  new 
analytical  problems  are  identified,  appropriate  investments  must  be  made  to  ensure  that  research  and 
development  is  planned  and  performed  to  define  and  develop  new  tools  and  techniques  to  address  the  ever- 
changing  landscape  of  military  operations  analysis. 

Expertise  location  includes  finding,  cataloging,  and  making  available  the  best  expertise  available  in  the 
DoD  components  when  needed  for  analysis  in  support  of  military  decisionmaking. 

Collaboration  enables  people  to  share  information,  expertise,  and  insights,  which  results  in  an 
amplification  of  tacit  knowledge,  which  in  turn,  results  in  enhanced  innovation  and  motivation. 

Knowledge  transfer  extends  the  reach  of  the  available  knowledge  and  skill  transfer  resources  to  remote 
locations,  enabling  virtual  teams  to  perform  at  high-level  organizational  standards,  independently  of  the 
geographical  location  of  team  members. 

Knowledge  management  is  a  new  discipline  and  is  receiving  fruitful,  innovative  tools  and  techniques 
from  the  research  community.  The  introduction  of  the  processes,  tools,  and  techniques  of  knowledge 
management  in  the  Military  Operations  Research  community  will  allow  us  to  be  more  effective  in  providing 
analysis  and  insight  on  important  issues  to  the  DoD  leadership. 
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APPENDIX  B:  KNOWLEDGE  MANAGEMENT  BIBLIOGRAPHY 

Books: 

Thomas  H.  Davenport  and  Laurance  Prusak,  Working  Knowledge:  How  Organizations  Manage  What  They 
Know,  Harvard  Business  School  Press,  1998 

Leif  Edvinsson  and  Michael  S.  Malone,  Intellectual  Capital:  Realizing  Your  Company’s  True  Value  by 
Finding  Its  Hidden  Brainpower,  HarperBusiness,  1997 

Arie  de  Geus,  The  Living  Company,  Harvard  Business  School  Press,  1997 

Dorothy  Leonard,  Wellsprings  of  Knowledge:  Building  and  Sustaining  the  Sources  of  Innovation,  Harvard 
Business  School  Press,  1995 

Ikujiro  Nonaka  and  Hirotaka  Takeuchi,  The  Knowledge  Creating  Company:  How  Japanese  Companies 
Create  the  Dynamic  of  Innovation,  Oxford  University  Press,  1995 

Thomas  A.  Stewart,  Intellectual  Capital:  The  New  Wealth  of  Organizations,  Doubleday,  1997 

Karl-Erik  Svieby,  The  New  Organizational  Wealth:  Managing  and  Measuring  Knowledge-Based  Assets, 
Berret-Koehler,  1997 

Lester  Thurow,  Building  Wealth:  The  New  Rules  for  Individuals,  Companies,  and  Nations  in  a  Knowledge- 
Based  Economy,  HarperCollins,  1999 


Gurus: 

John  Seely  Brown 

Chief  Scientist,  XEROX  and  Director  of  XEROX  PARC 
http://www.parc.xerox.com 

Ruth  David 

President  and  CEO,  ANSER 

Formerly:  Deputy  Director  for  Science  and  Technology,  CIA 
Implemented:  Agile  Intelligent  Enterprise  (AIE) 

Doug  Engelbart 

Stanford  Research  Institute,  Palo  Alto,  CA 
http://www.bootstrap.org 

Dorothy  Leonard 

Harvard  Business  School,  Cambridge,  MA 
http://www.harvard.edu 

Charles  Kirk 

American  Productivity  and  Quality  Center 
http://www.apqc.org/best/km/ 

Paul  Strassman 

http://www.strassmann.com/index.shtml 

Karl-Erik  Sveiby 

Sveiby  Knowledge  Management 

http://203.32.I0.69/default.htm 
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Daniel  Tkach, 

Technology  Director,  Institute  for  Knowledge  Management,  IBM 


High  Quality  Commercial  Vendors: 

Autonomy,  Inc. 
http://www.autonomy.com 

Instinctive  Technology,  Inc. 
http://www.instinctive.com 

International  Business  Machines,  Inc. 

Knowledge  Management  Consulting  and  Services  Practice 
Scott  J.  Smith,  smithjs@us.ibm.com 
http://www-4.ibm.com/software/data/know1edge/ 
http://www-3.ibm.com/solutions/businessintelligence/ 
http://www.software.ibm.com/data/km/knowledgex 

Inxight  Software  -  Spin-off  from  Xerox  PARC 
http://www.inxight.com 

Lotus  Development  Corporation,  Inc.  (owned  by  IBM) 
http://www.lotus.com/quickplace 

http://www.lotus.com/news/news.nsf/public/14F6B23E55BBC352852567B40Q628852 

Semio  Corporation,  Inc. 
http://www2.ideograf.com/semio/index.html 

Xerox  Corporation,  Inc. 
http://www.xerox.com/products/docushare 

Periodicals: 

Knowledge  Management  Magazine 
http://www.kmmag.com 

STRATEGY :  http://kmmag.com/kmmagn2/kml99908/featurec2.htm 

CULTURE:  http://kmmag.com/kminagn2/kml99908/featurec3.htm 

PROCESSES:  http://kmmag.com/kmmagn2/kml99908/featurec4.htm 
TECHNOLOGY:  http://kmmag.com/kmmagn2/kml999Q8/featurec5.htm 

KM  Implementations: 

Sugi  Sorenson 

Jet  Propulsion  Laboratories,  Pasadena,  CA 
Universities: 

Camegie-Mellon  University 

Center  for  Automated  Learning  and  Discovery 

Program  in  Knowledge  Discovery  and  Data  Mining 

http://www.cs.cmu.edu/~kdd/intro.html 

Harvard  University 
http  ://w  w  w .  harvard .  edu 
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Search  on:  knowledge  management 

Massachusetts  Institute  of  Technology 

http://www.mit.edu 

Search  on:  knowledge  management 

Other  Sources: 

US  Army  Web  Briefing  on  Knowledge  Management 
http://www.armv.mil/ako/WebBriefing/sld001.htm 

WWW  Virtual  Library  on  Knowledge  Management 
http://www.brint.com/km/ 

Special  Libraries  Association 
www.sla.org/membership/irc/knowledg.html 

Artificial  Intelligence  Applications  Institute 
www.aiai.ed.ac.uk/~alm/kam1nks.htm 


University  of  Colorado,  Denver 

www.cudenver.edu/~mryder/itc  data/org  learning.html 
Another  very  good  Bibliography 

http://www.irit.fr/ACTIVITES/EO  SMI/GRACO/biblio-ke.html 
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1000  Navy  Pentagon 

Washington  DC  20350-1000 

OFF:  (703)-602-31 79  DSN:  426-5223  FAX: 

Email:schmitz@hq.navy.mil 

DR  Jeffrey  E  Schofield 
Institute  for  Defense  Analyses 
IDA/OED 

1801  N.  Beauregard 
Alexandria  VA  2231 1-1772 
OFF:  (703)-845-6987  DSN:  FAX:  (703)-845-2274 
Email:jschofie@  ida.org 

PROF  David  A  Schrady  FS 
Naval  Postgraduate  School 
Dept  of  OR 
1  University  Circle 
Monterey  CA  93943-5001 

OFF:  831-656-2801  DSN:  878-2801  FAX:  831-656-2595 
Email:dschrady@nps.navy.mil 

MR  Jack  H.  Sheehan 

Applied  Research  Laboratories,  University  of  Texas 
c/o  DMSO 

1 901  N  Beauregard  St  #500 
Alexandria  VA  2231 1 

OFF:  703-824-3448  DSN:  FAX:  703-998-0667 
Email:jsheehan@dmso.mil 

DR  Robert  S  Sheldon 
S3I 

1700  Diagonal  Road  #500 
Alexandria  VA  22314 

OFF:  703-684-8268  DSN:  FAX:  703-684-8272 
Email:bs@s3i.com 

DR  Richard  C  Shirkey 

Army  Research  Lab 

Battlefield  Environment  Division  ATTN: 

AMSRL-IS-EW 

White  Sands  Missile  Range  NM  88002 

OFF:  505-678-5470  DSN:  258  FAX:  505-678-4449 

Email:rshirkey@arl.mil 
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MS  Elaine  R  Simmons 

OSD(PA&E)/JDS 

Crystal  Gateway  2,  Suite  200 

1225  Jefferson  Davis  Hwy 

Arlington  VA  22202-4301 

OFF:  703-41 4-1 904  DSN:  FAX:  703-41 4-81 1 4 

Email:elaine.simmons@osd.pentagon.mil 

Dr.  Richard  A  Siquig 
Naval  Research  Laboratory 
Code  7543 

7  Grace  Hopper  Ave,  Stop  2 
Monterey,  CA  93943-5502 
831-656-4732  FAX:  831-656-4769 
Email:  siquig@nrlmry.navy.mil 

PROF  Michael  G  Sovereign 
Naval  Postgraduate  School 
Code  IJWA 

589  Dyer  Road,  #200A 
Monterey  CA  93943-5000 

OFF:  831-656-3008  DSN:  878-3008  FAX:  831-656-3679 
Email:msovereign@  nps.navy.mil 

DR  Stuart  H  Starr  FS 
The  MITRE  Corporation 
1820  Dolley  Madison  Blvd  #MS  W557 
McLean  VA  221 02-3481 

OFF:  (703)-883-5494  DSN:  FAX:  (703)-883-1 373 
Email:starr@mitre.org 

MR  Joseph  S  Stewart 

The  MITRE  Corporation 

3504  Lake  Lynda  Drive  #1 55 

Orlando  FL  3281 7-1 472 

OFF:  407-208-1320  DSN:  FAX:  407-208-1320 

Email  :jss  @  .m  itre.org 

DR  Georgeanne  B.  Thibault 
ANSER 

1215  Jefferson  Davis  Hwy,  Suite  800 
Arlington,  VA  22202 
OFF:  703-416-3345  FAX  703-416-3329 
Email:  thibaulg@anser.org 

PROF  Alan  R  Washburn 

Naval  Postgraduate  School 

Code  ORA/VS 

Monterey  CA  93943-5000 

OFF:  (408)-656-31 27  DSN:  878-3127  FAX: 

Email:washburn@nps.navy.mil 


MR  Charles  P  Werchado 
OPNAV  N81 
2000  Navy  Pentagon 
Arlington  VA  20350 

OFF:  703-697-0482  DSN:  224-9131  FAX:  703-697-0738 
Email:werchado.chuck@hq.navy.mil 

MR  Richard  I  Wiles 
Military  Operations  Research  Society 
101  S  Whiting  St  #202 
Alexandria  VA  22304 

OFF:  (703)-751  -7290  DSN:  FAX:  (703)-751-8171 
Email:morsone@aol.com 

MR  James  L  Wilmeth  III 
Logicon 

J-8  The  Joint  Staff 
McLean  VA  2031 8 

OFF:  (703)-695-4657  DSN:  225-4657  FAX:  703-695-8031 
Emai!:wilmetjl@  js.pentagon.mil 

Mr  Joseph  Paul  Wilusz 
OSD(PA&E)  JDS 
1225  Jefferson  David  Highway 
Crystal  Gateway  II,  Suite  300 
Arlington  VA  22202 

OFF:  703-414-1959  DSN:  FAX:  703-414-8114 
Email:joseph-paul.wilusz@osd.pentagon.mil 

LTC  Mark  S  Woempner 
TRAC-WSMR 
Bldg  1400 

White  Sands  Missile  Range  NM  88002 

OFF:  505-678-2956  DSN:  258  FAX:  505-678-1450 

Email:woempnem@trac.  wsmr.army.mil 

LTC  Willis  Addison  Woods  Jr 
Strategic  and  Advanced  Computing  Center, 

ODISC4 
Building  323 
10125  Beach  Road 
Fort  Belvoir  VA  22060 

OFF:  703-704-3624  DSN:  224-1886  FAX:  (703)-693-5388 
Email:woodswa@hqda.army.mil 

Mr.  Mike  Yoemans 

ASD(C31)  Director  for  Knowledge 

Management 

Crystal  Gateway  2,  Suite  910 
1225  Jefferson  Davis  Highway 
Arlington  VA  22202 
OFF:  703-604-1474  DSN:  FAX: 
Email:yoemansM@osd.pentagon.mil 
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LTC  Kirk  A.  Yost 
The  Joint  Staff,  J8 
Washington,  DC  20318-8000 
OFF:  703-614-9751  FAX  703-693-5182 
Email:  yostka@js.pentagon.mil 

DR  MarkAYoungren 
MITRE 

Mail  Stop  W625 

1820  Dolley  Madison  Blvd. 

McLean  VA  221 02 

OFF:  703-883-6446  DSN:  FAX:  703-883-1370 
Emaikyoungren  @  mitre.org 


MAJ  Daniel  J  Zalewski 
OSD  (PA&E) 

1800  Defense  Pentagon 

Washington  DC  20301-1800 

OFF:  703-696-9370  DSN:  426  FAX:  703-696-9394 

Email:Daniel.Zalewski@osd.pentagon.mil 
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